Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision.

Voxel-wise visual encoding models based on convolutional neural networks (CNNs) have emerged as one of the prominent predictive tools of human brain activity via functional magnetic resonance imaging (fMRI) signals. While CNN-based models imitate the hierarchical structure of the human visual cortex to generate explainable features in response to natural visual stimuli, there is still a need for a brain-inspired model to predict brain responses accurately based on biomedical data. To bridge this gap, we propose a response prediction module called the Structurally Constrained Multi-Output (SCMO) module to include homologous correlations that arise between a group of voxels in a cortical region and predict more accurate responses. This module employs all the responses across a visual area to predict individual voxel-wise BOLD responses and therefore accounts for the population activity and collective behavior of voxels. Such a module can determine the relationships within each visual region by creating a structure matrix that represents the underlying voxel-to-voxel interactions. Moreover, since each response module in visual encoding tasks relies on the image features, we conducted experiments using two different feature extraction modules to assess the predictive performance of our proposed module. Specifically, we employed a recurrent CNN that integrates both feedforward and recurrent interactions, as well as the popular AlexNet model that utilizes feedforward connections. In the end, we demonstrate that the proposed framework provides a reliable predictive ability to generate brain responses across multiple areas, outperforming benchmark models in terms of stability and coherency of features.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus.

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others' social interactions. However, despite the prominence of social interactions in memory, the neural basis of remembering social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others' social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that a subset of the STS activity evoked by viewing social interactions predicted neural responses in not only held-out movie data, but also during memory recall. These results provide the first evidence that activity in the STS is reinstated in response to specific social content and that its reactivation underlies our ability to remember others' interactions. These findings further suggest that the STS contains representations of social interactions that are not only perceptually driven, but also more abstract or conceptual in nature.

Concurrent attention to hetero-depth surfaces in 3-D visual space is governed by theta rhythm.

When simultaneously confronted with multiple attentional targets, visual system employs a time-multiplexing approach in which each target alternates for prioritized access, a mechanism broadly known as rhythmic attentional sampling. For the past decade, rhythmic attentional sampling has received mounting support from converging behavioral and neural findings. However, so compelling are these findings that a critical test ground has been long overshadowed, namely the 3-D visual space where attention is complicated by extraction of the spatial layout of surfaces extending beyond 2-D planes. It remains unknown how attentional deployment to multiple targets is accomplished in the 3-D space. Here, we provided a time-resolved portrait of the behavioral and neural dynamics when participants concurrently attended to two surfaces defined by motion-depth conjunctions. To characterize the moment-to-moment attentional modulation effects, we measured perceptual sensitivity to the hetero-depth surface motions on a fine temporal scale and reconstructed their neural representations using a time-resolved multivariate inverted encoding model. We found that the perceptual sensitivity to the two surface motions rhythmically fluctuated over time at ~4 Hz, with one's enhancement closely tracked by the other's diminishment. Moreover, the behavioral pattern was coupled with an ongoing periodic alternation in strength between the two surface motion representations in the same frequency. Together, our findings provide the first converging evidence of an attentional "pendulum" that rhythmically traverses different stereoscopic depth planes and are indicative of a ubiquitous attentional time multiplexor based on theta rhythm in the 3-D visual space.

Hierarchical organization of social action features along the lateral visual pathway.

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.

Encoding of continuous perceptual choices in human early visual cortex.

Introduction: Research on the neural mechanisms of perceptual decision-making has typically focused on simple categorical choices, say between two alternative motion directions. Studies on such discrete alternatives have often suggested that choices are encoded either in a motor-based or in an abstract, categorical format in regions beyond sensory cortex. Methods: In this study, we used motion stimuli that could vary anywhere between 0° and 360° to assess how the brain encodes choices for features that span the full sensory continuum. We employed a combination of neuroimaging and encoding models based on Gaussian process regression to assess how either stimuli or choices were encoded in brain responses. Results: We found that single-voxel tuning patterns could be used to reconstruct the trial-by-trial physical direction of motion as well as the participants' continuous choices. Importantly, these continuous choice signals were primarily observed in early visual areas. The tuning properties in this region generalized between choice encoding and stimulus encoding, even for reports that reflected pure guessing. Discussion: We found only little information related to the decision outcome in regions beyond visual cortex, such as parietal cortex, possibly because our task did not involve differential motor preparation. This could suggest that decisions for continuous stimuli take can place already in sensory brain regions, potentially using similar mechanisms to the sensory recruitment in visual working memory.

Adding Meaning to Memories: How Parietal Cortex Combines Semantic Content with Episodic Experience.

Neuroimaging studies of human memory have consistently found that univariate responses in parietal cortex track episodic experience with stimuli (whether stimuli are 'old' or 'new'). More recently, pattern-based fMRI studies have shown that parietal cortex also carries information about the semantic content of remembered experiences. However, it is not well understood how memory-based and content-based signals are integrated within parietal cortex. Here, in humans (males and females), we used voxel-wise encoding models and a recognition memory task to predict the fMRI activity patterns evoked by complex natural scene images based on (1) the episodic history and (2) the semantic content of each image. Models were generated and compared across distinct subregions of parietal cortex and for occipitotemporal cortex. We show that parietal and occipitotemporal regions each encode memory and content information, but they differ in how they combine this information. Among parietal subregions, angular gyrus was characterized by robust and overlapping effects of memory and content. Moreover, subject-specific semantic tuning functions revealed that successful recognition shifted the amplitude of tuning functions in angular gyrus but did not change the selectivity of tuning. In other words, effects of memory and content were additive in angular gyrus. This pattern of data contrasted with occipitotemporal cortex where memory and content effects were interactive: memory effects were preferentially expressed by voxels tuned to the content of a remembered image. Collectively, these findings provide unique insight into how parietal cortex combines information about episodic memory and semantic content.SIGNIFICANCE STATEMENT Neuroimaging studies of human memory have identified multiple brain regions that not only carry information about "whether" a visual stimulus is successfully recognized but also "what" the content of that stimulus includes. However, a fundamental and open question concerns how the brain integrates these two types of information (memory and content). Here, using a powerful combination of fMRI analysis methods, we show that parietal cortex, particularly the angular gyrus, robustly combines memory- and content-related information, but these two forms of information are represented via additive, independent signals. In contrast, memory effects in high-level visual cortex critically depend on (and interact with) content representations. Together, these findings reveal multiple and distinct ways in which the brain combines memory- and content-related information.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory.

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Dynamic neural reconstructions of attended object location and features using EEG.

Attention allows us to select relevant and ignore irrelevant information from our complex environments. What happens when attention shifts from one item to another? To answer this question, it is critical to have tools that accurately recover neural representations of both feature and location information with high temporal resolution. In the present study, we used human electroencephalography (EEG) and machine learning to explore how neural representations of object features and locations update across dynamic shifts of attention. We demonstrate that EEG can be used to create simultaneous time courses of neural representations of attended features (time point-by-time point inverted encoding model reconstructions) and attended location (time point-by-time point decoding) during both stable periods and across dynamic shifts of attention. Each trial presented two oriented gratings that flickered at the same frequency but had different orientations; participants were cued to attend one of them and on half of trials received a shift cue midtrial. We trained models on a stable period from Hold attention trials and then reconstructed/decoded the attended orientation/location at each time point on Shift attention trials. Our results showed that both feature reconstruction and location decoding dynamically track the shift of attention and that there may be time points during the shifting of attention when 1) feature and location representations become uncoupled and 2) both the previously attended and currently attended orientations are represented with roughly equal strength. The results offer insight into our understanding of attentional shifts, and the noninvasive techniques developed in the present study lend themselves well to a wide variety of future applications.NEW & NOTEWORTHY We used human EEG and machine learning to reconstruct neural response profiles during dynamic shifts of attention. Specifically, we demonstrated that we could simultaneously read out both location and feature information from an attended item in a multistimulus display. Moreover, we examined how that readout evolves over time during the dynamic process of attentional shifts. These results provide insight into our understanding of attention, and this technique carries substantial potential for versatile extensions and applications.

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex.

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

Single-shot spiral diffusion-weighted imaging at 7T using expanded encoding with compressed sensing.

PURPOSE: The expanded encoding model incorporates spatially- and time-varying field perturbations for correction during reconstruction. To date, these reconstructions have used the conjugate gradient method with early stopping used as implicit regularization. However, this approach is likely suboptimal for low-SNR cases like diffusion or high-resolution MRI. Here, we investigate the extent that ℓ 1 $$ {\ell}_1 $$ -wavelet regularization, or equivalently compressed sensing (CS), combined with expanded encoding improves trade-offs between spatial resolution, readout time and SNR for single-shot spiral DWI at 7T. The reconstructions were performed using our open-source graphics processing unit-enabled reconstruction toolbox, "MatMRI," that allows inclusion of the different components of the expanded encoding model, with or without CS. METHODS: In vivo accelerated single-shot spirals were acquired with five acceleration factors (R) (2×-6×) and three in-plane spatial resolutions (1.5, 1.3, and 1.1 mm). From the in vivo reconstructions, we estimated diffusion tensors and computed fractional anisotropy maps. Then, simulations were used to quantitatively investigate and validate the impact of CS-based regularization on image quality when compared to a known ground truth. RESULTS: In vivo reconstructions revealed improved image quality with retainment of small features when CS was used. Simulations showed that the joint use of the expanded encoding model and CS improves accuracy of image reconstructions (reduced mean-squared error) over the range of R investigated. CONCLUSION: The expanded encoding model and CS regularization are complementary tools for single-shot spiral diffusion MRI, which enables both higher spatial resolutions and higher R.

Optical Encoding Model Based on Orbital Angular Momentum Powered by Machine Learning.

Based on orbital angular momentum (OAM) properties of Laguerre-Gaussian beams LG(p,ℓ), a robust optical encoding model for efficient data transmission applications is designed. This paper presents an optical encoding model based on an intensity profile generated by a coherent superposition of two OAM-carrying Laguerre-Gaussian modes and a machine learning detection method. In the encoding process, the intensity profile for data encoding is generated based on the selection of p and ℓ indices, while the decoding process is performed using a support vector machine (SVM) algorithm. Two different decoding models based on an SVM algorithm are tested to verify the robustness of the optical encoding model, finding a BER =10-9 for 10.2 dB of signal-to-noise ratio in one of the SVM models.

Eye movements disrupt EEG alpha-band coding of behaviorally relevant and irrelevant spatial locations held in working memory.

Oscillations in the alpha frequency band (∼8-12 Hz) of the human electroencephalogram play an important role in supporting selective attention to visual items and maintaining their spatial locations in working memory (WM). Recent findings suggest that spatial information maintained in alpha is modulated by interruptions to continuous visual input, such that attention shifts, eye closure, and backward masking of the encoded item cause reconstructed representations of remembered locations to become degraded. Here, we investigated how another common visual disruption-eye movements-modulates reconstructions of behaviorally relevant and irrelevant item locations held in WM. Participants completed a delayed estimation task, where they encoded and recalled either the location or color of an object after a brief retention period. During retention, participants either fixated at the center or executed a sequence of eye movements. Electroencephalography (EEG) was recorded at the scalp and eye position was monitored with an eye tracker. Inverted encoding modeling (IEM) was applied to reconstruct location-selective responses across multiple frequency bands during encoding and retention. Location-selective responses were successfully reconstructed from alpha activity during retention where participants fixated at the center, but these reconstructions were disrupted during eye movements. Recall performance decreased during eye-movements conditions but remained largely intact, and further analyses revealed that under specific task conditions, it was possible to reconstruct retained location information from lower frequency bands (1-4 Hz) during eye movements. These results suggest that eye movements disrupt maintained spatial information in alpha in a manner consistent with other acute interruptions to continuous visual input, but this information may be represented in other frequency bands.NEW & NOTEWORTHY Neural oscillations in the alpha frequency band support selective attention to visual items and maintenance of their spatial locations in human working memory. Here, we investigate how eye movements disrupt representations of item locations held in working memory. Although it was not possible to recover item locations from alpha during eye movements, retained location information could be recovered from select lower frequency bands. This suggests that during eye movements, stored spatial information may be represented in other frequencies.

Artificial neural network modelling of the neural population code underlying mathematical operations.

Mathematical operations have long been regarded as a sparse, symbolic process in neuroimaging studies. In contrast, advances in artificial neural networks (ANN) have enabled extracting distributed representations of mathematical operations. Recent neuroimaging studies have compared distributed representations of the visual, auditory and language domains in ANNs and biological neural networks (BNNs). However, such a relationship has not yet been examined in mathematics. Here we hypothesise that ANN-based distributed representations can explain brain activity patterns of symbolic mathematical operations. We used the fMRI data of a series of mathematical problems with nine different combinations of operators to construct voxel-wise encoding/decoding models using both sparse operator and latent ANN features. Representational similarity analysis demonstrated shared representations between ANN and BNN, an effect particularly evident in the intraparietal sulcus. Feature-brain similarity (FBS) analysis served to reconstruct a sparse representation of mathematical operations based on distributed ANN features in each cortical voxel. Such reconstruction was more efficient when using features from deeper ANN layers. Moreover, latent ANN features allowed the decoding of novel operators not used during model training from brain activity. The current study provides novel insights into the neural code underlying mathematical thought.

Neural representations of competing stimuli along the dorsal and ventral visual pathways during binocular rivalry.

Binocular rivalry arises when two discrepant stimuli are simultaneously presented to different eyes, during which observers consciously experience vivid perceptual alternations without physical changes in visual inputs. Neural dynamics tracking such perceptual alternations have been identified at both early and late visual areas, leading to the fundamental debate concerning the primary neural substrate underlying binocular rivalry. One promising hypothesis that might reconcile these seemingly paradoxical findings is a gradual shift from interocular competition between monocular neurons to pattern competition among binocular neurons. Here, we examined this hypothesis by investigating how neural representations of rivalrous stimuli evolved along the visual pathway. We found that representations of the dominant and the suppressed stimuli initially co-existed in V1, which were enhanced and attenuated respectively in extrastriate visual areas. Notably, neural activity in V4 was dictated by the representation of the dominant stimulus, while the representation of the suppressed stimulus was only partially inhibited in dorsal areas V3A and MT+. Our findings revealed a progressive transition from the co-existing representations of the rivalrous inputs to the dictatorial representation of the dominant stimulus in the ventral pathway, and advocated different cortical evolutionary patterns of visual representations between the dorsal and the ventral pathways.

High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks.

Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance. The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model's performance in high-level visual areas. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas.

Visual Responses to Moving and Flashed Stimuli of Neurons in Domestic Pigeon (Columba livia domestica) Optic Tectum.

Birds can rapidly and accurately detect moving objects for better survival in complex environments. This visual ability may be attributed to the response properties of neurons in the optic tectum. However, it is unknown how neurons in the optic tectum respond differently to moving objects compared to static ones. To address this question, neuronal activities were recorded from domestic pigeon (Columba livia domestica) optic tectum, responsible for orienting to moving objects, and the responses to moving and flashed stimuli were compared. An encoding model based on the Generalized Linear Model (GLM) framework was established to explain the difference in neuronal responses. The experimental results showed that the first spike latency to moving stimuli was smaller than that to flashed ones and firing rate was higher. The model further implied the faster and stronger response to a moving target result from spatiotemporal integration process, corresponding to the spatially sequential activation of tectal neurons and the accumulation of information in time. This study provides direct electrophysiological evidence about the different tectal neuron responses to moving objects and flashed ones. The findings of this investigation increase our understanding of the motion detection mechanism of tectal neurons.

Processing of visual statistics of naturalistic videos in macaque visual areas V1 and V4.

Natural scenes are characterized by diverse image statistics, including various parameters of the luminance histogram, outputs of Gabor-like filters, and pairwise correlations between the filter outputs of different positions, orientations, and scales (Portilla-Simoncelli statistics). Some of these statistics capture the response properties of visual neurons. However, it remains unclear to what extent such statistics can explain neural responses to natural scenes and how neurons that are tuned to these statistics are distributed across the cortex. Using two-photon calcium imaging and an encoding-model approach, we addressed these issues in macaque visual areas V1 and V4. For each imaged neuron, we constructed an encoding model to mimic its responses to naturalistic videos. By extracting Portilla-Simoncelli statistics through outputs of both filters and filter correlations, and by computing an optimally weighted sum of these outputs, the model successfully reproduced responses in a subpopulation of neurons. We evaluated the selectivities of these neurons by quantifying the contributions of each statistic to visual responses. Neurons whose responses were mainly determined by Gabor-like filter outputs (low-level statistics) were abundant at most imaging sites in V1. In V4, the relative contribution of higher order statistics, such as cross-scale correlation, was increased. Preferred image statistics varied markedly across V4 sites, and the response similarity of two neurons at individual imaging sites gradually declined with increasing cortical distance. The results indicate that natural scene analysis progresses from V1 to V4, and neurons sharing preferred image statistics are locally clustered in V4.

Encoding Model for Continuous Motion-sensitive Neurons in the Intermediate and Deep Layers of the Pigeon Optic Tectum.

The intermediate and deep layers of the optic tectum (OT) contain neurons that are sensitive to small continuously moving targets. The sensitivity of these neurons to continuously moving targets suggests directed energy accumulation in the dendrite field of these neurons. Considering that the activation of a single dendrite can induce somatic spikes in vitro, we suggest the mechanism underlying the sequential probability activation of soma. The simulation model of these neurons constructed in combination with the above assumptions qualitatively reproduces the response characteristics of neurons to multi-sized stimuli and continuous sensitivity stimuli observed in physiological experiments. We used the characteristics of continuous motion-sensitive neurons that prefer long-lasting motion and single dendrite activation to induce somatic spikes as the entry point to construct the neuron encoding model. This model will enhance our understanding of the information-processing mechanism of the OT area of bird neurons in perceiving weak targets, and has important theoretical and practical significance for the construction of new brain-like algorithms.

Sensory Recruitment Revisited: Ipsilateral V1 Involved in Visual Working Memory.

The "sensory recruitment hypothesis" posits an essential role of sensory cortices in working memory, beyond the well-accepted frontoparietal areas. Yet, this hypothesis has recently been challenged. In the present study, participants performed a delayed orientation recall task while high-spatial-resolution 3 T functional magnetic resonance imaging (fMRI) signals were measured in posterior cortices. A multivariate inverted encoding model approach was used to decode remembered orientations based on blood oxygen level-dependent fMRI signals from visual cortices during the delay period. We found that not only did activity in the contralateral primary visual cortex (V1) retain high-fidelity representations of the visual stimuli, but activity in the ipsilateral V1 also contained such orientation tuning. Moreover, although the encoded tuning was faded in the contralateral V1 during the late delay period, tuning information in the ipsilateral V1 remained sustained. Furthermore, the ipsilateral representation was presented in secondary visual cortex (V2) as well, but not in other higher-level visual areas. These results thus supported the sensory recruitment hypothesis and extended it to the ipsilateral sensory areas, which indicated the distributed involvement of visual areas in visual working memory.

Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing.

Recognizing others' social interactions is a crucial human ability. Using simple stimuli, previous studies have shown that social interactions are selectively processed in the superior temporal sulcus (STS), but prior work with movies has suggested that social interactions are processed in the medial prefrontal cortex (mPFC), part of the theory of mind network. It remains unknown to what extent social interaction selectivity is observed in real world stimuli when controlling for other covarying perceptual and social information, such as faces, voices, and theory of mind. The current study utilizes a functional magnetic resonance imaging (fMRI) movie paradigm and advanced machine learning methods to uncover the brain mechanisms uniquely underlying naturalistic social interaction perception. We analyzed two publicly available fMRI datasets, collected while both male and female human participants (n = 17 and 18) watched two different commercial movies in the MRI scanner. By performing voxel-wise encoding and variance partitioning analyses, we found that broad social-affective features predict neural responses in social brain regions, including the STS and mPFC. However, only the STS showed robust and unique selectivity specifically to social interactions, independent from other covarying features. This selectivity was observed across two separate fMRI datasets. These findings suggest that naturalistic social interaction perception recruits dedicated neural circuity in the STS, separate from the theory of mind network, and is a critical dimension of human social understanding.