Multimodal investigations of emotional face processing and social trait judgment of faces.

Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

Recurrent Connections in the Primate Ventral Visual Stream Mediate a Trade-Off Between Task Performance and Network Size During Core Object Recognition.

The computational role of the abundant feedback connections in the ventral visual stream is unclear, enabling humans and nonhuman primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then develop methods that allow us to compare both CNNs and ConvRNNs to finely grained measurements of primate categorization behavior and neural response trajectories across thousands of stimuli. We find that high-performing ConvRNNs provide a better match to these data than feedforward networks of any depth, predicting the precise timings at which each stimulus is behaviorally decoded from neural activation patterns. Moreover, these ConvRNN circuits consistently produce quantitatively accurate predictions of neural dynamics from V4 and IT across the entire stimulus presentation. In fact, we find that the highest-performing ConvRNNs, which best match neural and behavioral data, also achieve a strong Pareto trade-off between task performance and overall network size. Taken together, our results suggest the functional purpose of recurrence in the ventral pathway is to fit a high-performing network in cortex, attaining computational power through temporal rather than spatial complexity.

A Computational Probe into the Behavioral and Neural Markers of Atypical Facial Emotion Processing in Autism.

Despite ample behavioral evidence of atypical facial emotion processing in individuals with autism spectrum disorder (ASD), the neural underpinnings of such behavioral heterogeneities remain unclear. Here, I have used brain-tissue mapped artificial neural network (ANN) models of primate vision to probe candidate neural and behavior markers of atypical facial emotion recognition in ASD at an image-by-image level. Interestingly, the image-level behavioral patterns of the ANNs better matched the neurotypical subjects 'behavior than those measured in ASD. This behavioral mismatch was most remarkable when the ANN behavior was decoded from units that correspond to the primate inferior temporal (IT) cortex. ANN-IT responses also explained a significant fraction of the image-level behavioral predictivity associated with neural activity in the human amygdala (from epileptic patients without ASD), strongly suggesting that the previously reported facial emotion intensity encodes in the human amygdala could be primarily driven by projections from the IT cortex. In sum, these results identify primate IT activity as a candidate neural marker and demonstrate how ANN models of vision can be used to generate neural circuit-level hypotheses and guide future human and nonhuman primate studies in autism.SIGNIFICANCE STATEMENT Moving beyond standard parametric approaches that predict behavior with high-level categorical descriptors of a stimulus (e.g., level of happiness/fear in a face image), in this study, I demonstrate how an image-level probe, using current deep-learning-based ANN models, allows identification of more diagnostic stimuli for autism spectrum disorder enabling the design of more powerful experiments. This study predicts that IT cortex activity is a key candidate neural marker of atypical facial emotion processing in people with ASD. Importantly, the results strongly suggest that ASD-related atypical facial emotion intensity encodes in the human amygdala could be primarily driven by projections from the IT cortex.

Fast Recurrent Processing via Ventrolateral Prefrontal Cortex Is Needed by the Primate Ventral Stream for Robust Core Visual Object Recognition.

Distributed neural population spiking patterns in macaque inferior temporal (IT) cortex that support core object recognition require additional time to develop for specific, "late-solved" images. This suggests the necessity of recurrent processing in these computations. Which brain circuits are responsible for computing and transmitting these putative recurrent signals to IT? To test whether the ventrolateral prefrontal cortex (vlPFC) is a critical recurrent node in this system, here, we pharmacologically inactivated parts of vlPFC and simultaneously measured IT activity while monkeys performed object discrimination tasks. vlPFC inactivation deteriorated the quality of late-phase (>150 ms from image onset) IT population code and produced commensurate behavioral deficits for late-solved images. Finally, silencing vlPFC caused the monkeys' IT activity and behavior to become more like those produced by feedforward-only ventral stream models. Together with prior work, these results implicate fast recurrent processing through vlPFC as critical to producing behaviorally sufficient object representations in IT.

An Open Resource for Non-human Primate Optogenetics.

Optogenetics has revolutionized neuroscience in small laboratory animals, but its effect on animal models more closely related to humans, such as non-human primates (NHPs), has been mixed. To make evidence-based decisions in primate optogenetics, the scientific community would benefit from a centralized database listing all attempts, successful and unsuccessful, of using optogenetics in the primate brain. We contacted members of the community to ask for their contributions to an open science initiative. As of this writing, 45 laboratories around the world contributed more than 1,000 injection experiments, including precise details regarding their methods and outcomes. Of those entries, more than half had not been published. The resource is free for everyone to consult and contribute to on the Open Science Framework website. Here we review some of the insights from this initial release of the database and discuss methodological considerations to improve the success of optogenetic experiments in NHPs.

The inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys.

The ability to recognize written letter strings is foundational to human reading, but the underlying neuronal mechanisms remain largely unknown. Recent behavioral research in baboons suggests that non-human primates may provide an opportunity to investigate this question. We recorded the activity of hundreds of neurons in V4 and the inferior temporal cortex (IT) while naïve macaque monkeys passively viewed images of letters, English words and non-word strings, and tested the capacity of those neuronal representations to support a battery of orthographic processing tasks. We found that simple linear read-outs of IT (but not V4) population responses achieved high performance on all tested tasks, even matching the performance and error patterns of baboons on word classification. These results show that the IT cortex of untrained primates can serve as a precursor of orthographic processing, suggesting that the acquisition of reading in humans relies on the recycling of a brain network evolved for other visual functions.

Transcranial alternating current stimulation attenuates BOLD adaptation and increases functional connectivity.

Transcranial alternating current stimulation (tACS) is used as a noninvasive tool for cognitive enhancement and clinical applications. The physiological effects of tACS, however, are complex and poorly understood. Most studies of tACS focus on its ability to entrain brain oscillations, but our behavioral results in humans and extracellular recordings in nonhuman primates support the view that tACS at 10 Hz also affects brain function by reducing sensory adaptation. Our primary goal in the present study is to test this hypothesis using blood oxygen level-dependent (BOLD) imaging in human subjects. Using concurrent functional magnetic resonance imaging (fMRI) and tACS, and a motion adaptation paradigm developed to quantify BOLD adaptation, we show that tACS significantly attenuates adaptation in the human motion area (hMT+). In addition, an exploratory analysis shows that tACS increases functional connectivity of the stimulated hMT+ with the rest of the brain and the dorsal attention network in particular. Based on field estimates from individualized head models, we relate these changes to the strength of tACS-induced electric fields. Specifically, we report that functional connectivity (between hMT+ and any other region of interest) increases in proportion to the field strength in the region of interest. These findings add support for the claim that weak 10-Hz currents applied to the scalp modulate both local and global measures of brain activity.NEW & NOTEWORTHY Concurrent transcranial alternating current stimulation (tACS) and functional MRI show that tACS affects the human brain by attenuating adaptation and increasing functional connectivity in a dose-dependent manner. This work is important for our basic understanding of what tACS does, but also for therapeutic applications, which need insight into the full range of ways in which tACS affects the brain.

Neural population control via deep image synthesis.

Particular deep artificial neural networks (ANNs) are today's most accurate models of the primate brain's ventral visual stream. Using an ANN-driven image synthesis method, we found that luminous power patterns (i.e., images) can be applied to primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. This method, although not yet perfect, achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping receptive fields. These results show how the knowledge embedded in today's ANN models might be used to noninvasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.

Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior.

Non-recurrent deep convolutional neural networks (CNNs) are currently the best at modeling core object recognition, a behavior that is supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. If recurrence is critical to this behavior, then primates should outperform feedforward-only deep CNNs for images that require additional recurrent processing beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these 'challenge' images. Second, using large-scale electrophysiology, we observed that behaviorally sufficient object identity solutions emerged ~30 ms later in the IT cortex for challenge images compared with primate performance-matched 'control' images. Third, these behaviorally critical late-phase IT response patterns were poorly predicted by feedforward deep CNN activations. Notably, very-deep CNNs and shallower recurrent CNNs better predicted these late IT responses, suggesting that there is a functional equivalence between additional nonlinear transformations and recurrence. Beyond arguing that recurrent circuits are critical for rapid object identification, our results provide strong constraints for future recurrent model development.

Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks.

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models.SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.

Transcranial Alternating Current Stimulation Attenuates Neuronal Adaptation.

We previously showed that brief application of 2 mA (peak-to-peak) transcranial currents alternating at 10 Hz significantly reduces motion adaptation in humans. This is but one of many behavioral studies showing that weak currents applied to the scalp modulate neural processing. Transcranial stimulation has been shown to improve perception, learning, and a range of clinical symptoms. Few studies, however, have measured the neural consequences of transcranial current stimulation. We capitalized on the strong link between motion perception and neural activity in the middle temporal (MT) area of the macaque monkey to study the neural mechanisms that underlie the behavioral consequences of transcranial alternating current stimulation. First, we observed that 2 mA currents generated substantial intracranial fields, which were much stronger in the stimulated hemisphere (0.12 V/m) than on the opposite side of the brain (0.03 V/m). Second, we found that brief application of transcranial alternating current stimulation at 10 Hz reduced spike-frequency adaptation of MT neurons and led to a broadband increase in the power spectrum of local field potentials. Together, these findings provide a direct demonstration that weak electric fields applied to the scalp significantly affect neural processing in the primate brain and that this includes a hitherto unknown mechanism that attenuates sensory adaptation.SIGNIFICANCE STATEMENT Transcranial stimulation has been claimed to improve perception, learning, and a range of clinical symptoms. Little is known, however, how transcranial current stimulation generates such effects, and the search for better stimulation protocols proceeds largely by trial and error. We investigated, for the first time, the neural consequences of stimulation in the monkey brain. We found that even brief application of alternating current stimulation reduced the effects of adaptation on single-neuron firing rates and local field potentials; this mechanistic insight explains previous behavioral findings and suggests a novel way to modulate neural information processing using transcranial currents. In addition, by developing an animal model to help understand transcranial stimulation, this study will aid the rational design of stimulation protocols for the treatment of mental illnesses, and the improvement of perception and learning.

Testing the assumptions underlying fMRI adaptation using intracortical recordings in area MT.

We investigated how neural activity in the middle temporal area of the macaque monkey changes after 3 sec of exposure to a visual stimulus and used this to gain insight into the assumptions underlying the fMRI adaptation method (fMRIa). We studied both changes in tuning curves following weak and strong motion stimuli (adaptation) and the differences between a first and second exposure to the same stimulus (repetition suppression). Typically, tuning curves had smaller amplitudes and narrower tuning widths after strong adaptation; this was true for single neurons, multi-unit activity (MUA), the evoked local field potential (LFP), as well as gamma band activity. Repetition typically led to reduced responses. This reduction was correlated with direction selectivity and not explained by neural fatigue. Our data, however, warn against a simplistic view of the consequences of adaptation. First, a considerable fraction of neurons and sites showed response enhancements after adaptation, especially when probed with a stimulus that moved opposite to the direction of the adapting stimulus. Second, adaptation was stimulus selective only on a time scale of ∼100 msec. Third, aggregate measures of neural activity (MUA, LFPs) had substantially different adaptation effects. Fourth, there were qualitative differences between our findings in MT and earlier findings in IT cortex. We conclude that selective adaptation effects in fMRIa are relatively easy to miss even when they exist (for instance by presenting stimuli for too long, or because neurons that enhance after adaptation cancel out the effect of neurons that suppress). Moreover, we argue that adaptation should be understood in the context of the computations that a neural circuit perform. Using fMRIa as a tool to uncover neural selectivity requires a better understanding of this circuitry and its consequences for adaptation.

Social closeness and feedback modulate susceptibility to the framing effect.

Although we often seek social feedback (SFB) from others to help us make decisions, little is known about how SFB affects decisions under risk, particularly from a close peer. We conducted two experiments using an established framing task to probe how decision-making is modulated by SFB valence (positive, negative) and the level of closeness with feedback provider (friend, confederate). Participants faced mathematically equivalent decisions framed as either an opportunity to keep (gain frame) or lose (loss frame) part of an initial endowment. Periodically, participants were provided with positive (e.g., "Nice!") or negative (e.g., "Lame!") feedback about their choices. Such feedback was provided by either a confederate (Experiment 1) or a gender-matched close friend (Experiment 2). As expected, the framing effect was observed in both experiments. Critically, an individual's susceptibility to the framing effect was modulated by the valence of the SFB, but only when the feedback provider was a close friend. This effect was reflected in the activation patterns of ventromedial prefrontal cortex and posterior cingulate cortex, regions involved in complex decision-making. Taken together, these results highlight social closeness as an important factor in understanding the impact of SFB on neural mechanisms of decision-making.

Transcranial alternating current stimulation attenuates visual motion adaptation.

Transcranial alternating current stimulation (tACS) is used in clinical applications and basic neuroscience research. Although its behavioral effects are evident from prior reports, current understanding of the mechanisms that underlie these effects is limited. We used motion perception, a percept with relatively well known properties and underlying neural mechanisms to investigate tACS mechanisms. Healthy human volunteers showed a surprising improvement in motion sensitivity when visual stimuli were paired with 10 Hz tACS. In addition, tACS reduced the motion-after effect, and this reduction was correlated with the improvement in motion sensitivity. Electrical stimulation had no consistent effect when applied before presenting a visual stimulus or during recovery from motion adaptation. Together, these findings suggest that perceptual effects of tACS result from an attenuation of adaptation. Important consequences for the practical use of tACS follow from our work. First, because this mechanism interferes only with adaptation, this suggests that tACS can be targeted at subsets of neurons (by adapting them), even when the applied currents spread widely throughout the brain. Second, by interfering with adaptation, this mechanism provides a means by which electrical stimulation can generate behavioral effects that outlast the stimulation.

Probing the mechanisms underlying the mitigation of cognitive aging with anodal transcranial direct current stimulation.

Meinzer et al. (J Neurosci 33: 12470-12478, 2013) have recently reported that anodal transcranial direct current stimulation (atDCS) mitigates age-related cognitive changes. Simultaneous measurement of BOLD signal during atDCS also showed "youth-like" processing in an elderly population. Although the effects are very promising, the underlying mechanisms of atDCS are still not clear. In this article, we provide a critical review of the results, emphasizing the article's significance and providing additional insight that will help elucidate the results and atDCS mechanisms.

Transcranial electrical stimulation over visual cortex evokes phosphenes with a retinal origin.

Transcranial electrical stimulation (tES) is a promising therapeutic tool for a range of neurological diseases. Understanding how the small currents used in tES spread across the scalp and penetrate the brain will be important for the rational design of tES therapies. Alternating currents applied transcranially above visual cortex induce the perception of flashes of light (phosphenes). This makes the visual system a useful model to study tES. One hypothesis is that tES generates phosphenes by direct stimulation of the cortex underneath the transcranial electrode. Here, we provide evidence for the alternative hypothesis that phosphenes are generated in the retina by current spread from the occipital electrode. Building on the existing literature, we first confirm that phosphenes are induced at lower currents when electrodes are placed farther away from visual cortex and closer to the eye. Second, we explain the temporal frequency tuning of phosphenes based on the well-known response properties of primate retinal ganglion cells. Third, we show that there is no difference in the time it takes to evoke phosphenes in the retina or by stimulation above visual cortex. Together, these findings suggest that phosphenes induced by tES over visual cortex originate in the retina. From this, we infer that tES currents spread well beyond the area of stimulation and are unlikely to lead to focal neural activation. Novel stimulation protocols that optimize current distributions are needed to overcome these limitations of tES.