The Medusa effect reveals levels of mind perception in pictures.

Throughout our species history, humans have created pictures. The resulting picture record reveals an overwhelming preference for depicting things with minds. This preference suggests that pictures capture something of the mind that is significant to us, albeit at reduced potency. Here, we show that abstraction dims the perceived mind, even within the same picture. In a series of experiments, people were perceived as more real, and higher in both Agency (ability to do) and Experience (ability to feel), when they were presented as pictures than when they were presented as pictures of pictures. This pattern persisted across different tasks and even when comparators were matched for identity and image size. Viewers spontaneously discriminated between different levels of abstraction during eye tracking and were less willing to share money with a more abstracted person in a dictator game. Given that mind perception underpins moral judgement, our findings suggest that depicted persons will receive greater or lesser ethical consideration, depending on the level of abstraction.

Drifting codes within a stable coding scheme for working memory.

Working memory (WM) is important to maintain information over short time periods to provide some stability in a constantly changing environment. However, brain activity is inherently dynamic, raising a challenge for maintaining stable mental states. To investigate the relationship between WM stability and neural dynamics, we used electroencephalography to measure the neural response to impulse stimuli during a WM delay. Multivariate pattern analysis revealed representations were both stable and dynamic: there was a clear difference in neural states between time-specific impulse responses, reflecting dynamic changes, yet the coding scheme for memorised orientations was stable. This suggests that a stable subcomponent in WM enables stable maintenance within a dynamic system. A stable coding scheme simplifies readout for WM-guided behaviour, whereas the low-dimensional dynamic component could provide additional temporal information. Despite having a stable subspace, WM is clearly not perfect-memory performance still degrades over time. Indeed, we find that even within the stable coding scheme, memories drift during maintenance. When averaged across trials, such drift contributes to the width of the error distribution.

Neurophysiological evidence for crossmodal (face-name) person-identity representation in the human left ventral temporal cortex.

Putting a name to a face is a highly common activity in our daily life that greatly enriches social interactions. Although this specific person-identity association becomes automatic with learning, it remains difficult and can easily be disrupted in normal circumstances or neurological conditions. To shed light on the neural basis of this important and yet poorly understood association between different input modalities in the human brain, we designed a crossmodal frequency-tagging paradigm coupled to brain activity recording via scalp and intracerebral electroencephalography. In Experiment 1, 12 participants were presented with variable pictures of faces and written names of a single famous identity at a 4-Hz frequency rate while performing an orthogonal task. Every 7 items, another famous identity appeared, either as a face or a name. Robust electrophysiological responses were found exactly at the frequency of identity change (i.e., 4 Hz / 7 = 0.571 Hz), suggesting a crossmodal neural response to person identity. In Experiment 2 with twenty participants, two control conditions with periodic changes of identity for faces or names only were added to estimate the contribution of unimodal neural activity to the putative crossmodal face-name responses. About 30% of the response occurring at the frequency of crossmodal identity change over the left occipito-temporal cortex could not be accounted for by the linear sum of unimodal responses. Finally, intracerebral recordings in the left ventral anterior temporal lobe (ATL) in 7 epileptic patients tested with this paradigm revealed a small number of "pure" crossmodal responses, i.e., with no response to changes of identity for faces or names only. Altogether, these observations provide evidence for integration of verbal and nonverbal person identity-specific information in the human brain, highlighting the contribution of the left ventral ATL in the automatic retrieval of face-name identity associations.

Context effects on probability estimation.

Many decisions rely on how we evaluate potential outcomes and estimate their corresponding probabilities of occurrence. Outcome evaluation is subjective because it requires consulting internal preferences and is sensitive to context. In contrast, probability estimation requires extracting statistics from the environment and therefore imposes unique challenges to the decision maker. Here, we show that probability estimation, like outcome evaluation, is subject to context effects that bias probability estimates away from other events present in the same context. However, unlike valuation, these context effects appeared to be scaled by estimated uncertainty, which is largest at intermediate probabilities. Blood-oxygen-level-dependent (BOLD) imaging showed that patterns of multivoxel activity in the dorsal anterior cingulate cortex (dACC), ventromedial prefrontal cortex (VMPFC), and intraparietal sulcus (IPS) predicted individual differences in context effects on probability estimates. These results establish VMPFC as the neurocomputational substrate shared between valuation and probability estimation and highlight the additional involvement of dACC and IPS that can be uniquely attributed to probability estimation. Because probability estimation is a required component of computational accounts from sensory inference to higher cognition, the context effects found here may affect a wide array of cognitive computations.

Adaptive social networks promote the wisdom of crowds.

Social networks continuously change as new ties are created and existing ones fade. It is widely acknowledged that our social embedding has a substantial impact on what information we receive and how we form beliefs and make decisions. However, most empirical studies on the role of social networks in collective intelligence have overlooked the dynamic nature of social networks and its role in fostering adaptive collective intelligence. Therefore, little is known about how groups of individuals dynamically modify their local connections and, accordingly, the topology of the network of interactions to respond to changing environmental conditions. In this paper, we address this question through a series of behavioral experiments and supporting simulations. Our results reveal that, in the presence of plasticity and feedback, social networks can adapt to biased and changing information environments and produce collective estimates that are more accurate than their best-performing member. To explain these results, we explore two mechanisms: 1) a global-adaptation mechanism where the structural connectivity of the network itself changes such that it amplifies the estimates of high-performing members within the group (i.e., the network "edges" encode the computation); and 2) a local-adaptation mechanism where accurate individuals are more resistant to social influence (i.e., adjustments to the attributes of the "node" in the network); therefore, their initial belief is disproportionately weighted in the collective estimate. Our findings substantiate the role of social-network plasticity and feedback as key adaptive mechanisms for refining individual and collective judgments.

The interrelationship between the face and vocal tract configuration during audiovisual speech.

It is well established that speech perception is improved when we are able to see the speaker talking along with hearing their voice, especially when the speech is noisy. While we have a good understanding of where speech integration occurs in the brain, it is unclear how visual and auditory cues are combined to improve speech perception. One suggestion is that integration can occur as both visual and auditory cues arise from a common generator: the vocal tract. Here, we investigate whether facial and vocal tract movements are linked during speech production by comparing videos of the face and fast magnetic resonance (MR) image sequences of the vocal tract. The joint variation in the face and vocal tract was extracted using an application of principal components analysis (PCA), and we demonstrate that MR image sequences can be reconstructed with high fidelity using only the facial video and PCA. Reconstruction fidelity was significantly higher when images from the two sequences corresponded in time, and including implicit temporal information by combining contiguous frames also led to a significant increase in fidelity. A "Bubbles" technique was used to identify which areas of the face were important for recovering information about the vocal tract, and vice versa, on a frame-by-frame basis. Our data reveal that there is sufficient information in the face to recover vocal tract shape during speech. In addition, the facial and vocal tract regions that are important for reconstruction are those that are used to generate the acoustic speech signal.

Decomposing loss aversion from gaze allocation and pupil dilation.

Loss-averse decisions, in which one avoids losses at the expense of gains, are highly prevalent. However, the underlying mechanisms remain controversial. The prevailing account highlights a valuation bias that overweighs losses relative to gains, but an alternative view stresses a response bias to avoid choices involving potential losses. Here we couple a computational process model with eye-tracking and pupillometry to develop a physiologically grounded framework for the decision process leading to accepting or rejecting gambles with equal odds of winning and losing money. Overall, loss-averse decisions were accompanied by preferential gaze toward losses and increased pupil dilation for accepting gambles. Using our model, we found gaze allocation selectively indexed valuation bias, and pupil dilation selectively indexed response bias. Finally, we demonstrate that our computational model and physiological biomarkers can identify distinct types of loss-averse decision makers who would otherwise be indistinguishable using conventional approaches. Our study provides an integrative framework for the cognitive processes that drive loss-averse decisions and highlights the biological heterogeneity of loss aversion across individuals.

Acoustic information about upper limb movement in voicing.

We show that the human voice has complex acoustic qualities that are directly coupled to peripheral musculoskeletal tensioning of the body, such as subtle wrist movements. In this study, human vocalizers produced a steady-state vocalization while rhythmically moving the wrist or the arm at different tempos. Although listeners could only hear and not see the vocalizer, they were able to completely synchronize their own rhythmic wrist or arm movement with the movement of the vocalizer which they perceived in the voice acoustics. This study corroborates recent evidence suggesting that the human voice is constrained by bodily tensioning affecting the respiratory-vocal system. The current results show that the human voice contains a bodily imprint that is directly informative for the interpersonal perception of another's dynamic physical states.

Stable maintenance of multiple representational formats in human visual short-term memory.

Visual short-term memory (VSTM) enables humans to form a stable and coherent representation of the external world. However, the nature and temporal dynamics of the neural representations in VSTM that support this stability are barely understood. Here we combined human intracranial electroencephalography (iEEG) recordings with analyses using deep neural networks and semantic models to probe the representational format and temporal dynamics of information in VSTM. We found clear evidence that VSTM maintenance occurred in two distinct representational formats which originated from different encoding periods. The first format derived from an early encoding period (250 to 770 ms) corresponded to higher-order visual representations. The second format originated from a late encoding period (1,000 to 1,980 ms) and contained abstract semantic representations. These representational formats were overall stable during maintenance, with no consistent transformation across time. Nevertheless, maintenance of both representational formats showed substantial arrhythmic fluctuations, i.e., waxing and waning in irregular intervals. The increases of the maintained representational formats were specific to the phases of hippocampal low-frequency activity. Our results demonstrate that human VSTM simultaneously maintains representations at different levels of processing, from higher-order visual information to abstract semantic representations, which are stably maintained via coupling to hippocampal low-frequency activity.

Hippocampal theta codes for distances in semantic and temporal spaces.

The medial temporal lobe (MTL) is known to support episodic memory and spatial navigation, raising the possibility that its true function is to form "cognitive maps" of any kind of information. Studies in humans and animals support the idea that the hippocampal theta rhythm (4 to 8 Hz) is key to this mapping function, as it has been repeatedly observed during spatial navigation tasks. If episodic memory and spatial navigation are 2 sides of the same coin, we hypothesized that theta oscillations might reflect relations between explicitly nonspatial items, such as words. We asked 189 neurosurgical patients to perform a verbal free-recall task, of which 96 had indwelling electrodes placed in the MTL. Subjects were instructed to remember short lists of sequentially presented nouns. We found that hippocampal theta power and connectivity during item retrieval coded for semantic distances between words, as measured using word2vec-derived subspaces. Additionally, hippocampal theta indexed temporal distances between words after filtering lists on recall performance, to ensure adequate dynamic range in time. Theta effects were noted only for semantic subspaces of 1 dimension, indicating a substantial compression of the possible semantic feature space. These results lend further support to our growing confidence that the MTL forms cognitive maps of arbitrary representational spaces, helping to reconcile longstanding differences between the spatial and episodic memory literatures.

Integration of speed and time for estimating time to contact.

To coordinate movements with events in a dynamic environment the brain has to anticipate when those events occur. A classic example is the estimation of time to contact (TTC), that is, when an object reaches a target. It is thought that TTC is estimated from kinematic variables. For example, a tennis player might use an estimate of distance (d) and speed (v) to estimate TTC (TTC = d/v). However, the tennis player may instead estimate TTC as twice the time it takes for the ball to move from the serve line to the net line. This latter strategy does not rely on kinematics and instead computes TTC solely from temporal cues. Which of these two strategies do humans use to estimate TTC? Considering that both speed and time estimates are inherently uncertain and the ability of the human brain to combine different sources of information, we hypothesized that humans estimate TTC by integrating speed information with temporal cues. We evaluated this hypothesis systematically using psychophysics and Bayesian modeling. Results indicated that humans rely on both speed information and temporal cues and integrate them to optimize their TTC estimates when both cues are present. These findings suggest that the brain's timing mechanisms are actively engaged when interacting with dynamic stimuli.

Tactile distance illusions reflect a coherent stretch of tactile space.

Illusions of the perception of distance between two touches on the skin have been described since the classic work of Weber in the 19th century. The perceptual mechanisms underlying such spatial distortions, however, remain poorly understood. One potential interpretation is that the representational space of touch is related to the true structure of the skin by a geometrically simple stretch. If distortions of tactile distance perception reflect a simple stretch of tactile space, perceived distance should vary predictably as a function of the orientation of the stimulus on the skin, showing a sinusoidal pattern. Here, we tested this prediction by obtained judgments of perceived tactile distance for pairs of touches aligned with eight orientations on the skin. Across four experiments, the results were highly consistent with this prediction, showing no apparent deviation from a model of simple stretch of tactile space. Similar results were apparent on both the dorsum and palm of the hand, as well as the forehead. These results show that spatial distortions of touch are well characterized by a geometrically simple stretch of tactile space.

How cognitive and reactive fear circuits optimize escape decisions in humans.

Flight initiation distance (FID), the distance at which an organism flees from an approaching threat, is an ecological metric of cost-benefit functions of escape decisions. We adapted the FID paradigm to investigate how fast- or slow-attacking "virtual predators" constrain escape decisions. We show that rapid escape decisions rely on "reactive fear" circuits in the periaqueductal gray and midcingulate cortex (MCC), while protracted escape decisions, defined by larger buffer zones, were associated with "cognitive fear" circuits, which include posterior cingulate cortex, hippocampus, and the ventromedial prefrontal cortex, circuits implicated in more complex information processing, cognitive avoidance strategies, and behavioral flexibility. Using a Bayesian decision-making model, we further show that optimization of escape decisions under rapid flight were localized to the MCC, a region involved in adaptive motor control, while the hippocampus is implicated in optimizing decisions that update and control slower escape initiation. These results demonstrate an unexplored link between defensive survival circuits and their role in adaptive escape decisions.

Frequency modulation of neural oscillations according to visual task demands.

Temporal integration in visual perception is thought to occur within cycles of occipital alpha-band (8-12 Hz) oscillations. Successive stimuli may be integrated when they fall within the same alpha cycle and segregated for different alpha cycles. Consequently, the speed of alpha oscillations correlates with the temporal resolution of perception, such that lower alpha frequencies provide longer time windows for perceptual integration and higher alpha frequencies correspond to faster sampling and segregation. Can the brain's rhythmic activity be dynamically controlled to adjust its processing speed according to different visual task demands? We recorded magnetoencephalography (MEG) while participants switched between task instructions for temporal integration and segregation, holding stimuli and task difficulty constant. We found that the peak frequency of alpha oscillations decreased when visual task demands required temporal integration compared with segregation. Alpha frequency was strategically modulated immediately before and during stimulus processing, suggesting a preparatory top-down source of modulation. Its neural generators were located in occipital and inferotemporal cortex. The frequency modulation was specific to alpha oscillations and did not occur in the delta (1-3 Hz), theta (3-7 Hz), beta (15-30 Hz), or gamma (30-50 Hz) frequency range. These results show that alpha frequency is under top-down control to increase or decrease the temporal resolution of visual perception.

Visual perception as retrospective Bayesian decoding from high- to low-level features.

When a stimulus is presented, its encoding is known to progress from low- to high-level features. How these features are decoded to produce perception is less clear, and most models assume that decoding follows the same low- to high-level hierarchy of encoding. There are also theories arguing for global precedence, reversed hierarchy, or bidirectional processing, but they are descriptive without quantitative comparison with human perception. Moreover, observers often inspect different parts of a scene sequentially to form overall perception, suggesting that perceptual decoding requires working memory, yet few models consider how working-memory properties may affect decoding hierarchy. We probed decoding hierarchy by comparing absolute judgments of single orientations and relative/ordinal judgments between two sequentially presented orientations. We found that lower-level, absolute judgments failed to account for higher-level, relative/ordinal judgments. However, when ordinal judgment was used to retrospectively decode memory representations of absolute orientations, striking aspects of absolute judgments, including the correlation and forward/backward aftereffects between two reported orientations in a trial, were explained. We propose that the brain prioritizes decoding of higher-level features because they are more behaviorally relevant, and more invariant and categorical, and thus easier to specify and maintain in noisy working memory, and that more reliable higher-level decoding constrains less reliable lower-level decoding.

Motor origin of temporal predictions in auditory attention.

In behavior, action and perception are inherently interdependent. However, the actual mechanistic contributions of the motor system to sensory processing are unknown. We present neurophysiological evidence that the motor system is involved in predictive timing, a brain function that aligns temporal fluctuations of attention with the timing of events in a task-relevant stream, thus facilitating sensory selection and optimizing behavior. In a magnetoencephalography experiment involving auditory temporal attention, participants had to disentangle two streams of sound on the unique basis of endogenous temporal cues. We show that temporal predictions are encoded by interdependent delta and beta neural oscillations originating from the left sensorimotor cortex, and directed toward auditory regions. We also found that overt rhythmic movements improved the quality of temporal predictions and sharpened the temporal selection of relevant auditory information. This latter behavioral and functional benefit was associated with increased signaling of temporal predictions in right-lateralized frontoparietal associative regions. In sum, this study points at a covert form of auditory active sensing. Our results emphasize the key role of motor brain areas in providing contextual temporal information to sensory regions, driving perceptual and behavioral selection.

Rethinking people's conceptions of mental life.

How do people make sense of the emotions, sensations, and cognitive abilities that make up mental life? Pioneering work on the dimensions of mind perception has been interpreted as evidence that people consider mental life to have two core components-experience (e.g., hunger, joy) and agency (e.g., planning, self-control) [Gray HM, et al. (2007) Science 315:619]. We argue that this conclusion is premature: The experience-agency framework may capture people's understanding of the differences among different beings (e.g., dogs, humans, robots, God) but not how people parse mental life itself. Inspired by Gray et al.'s bottom-up approach, we conducted four large-scale studies designed to assess people's conceptions of mental life more directly. This led to the discovery of an organization that differs strikingly from the experience-agency framework: Instead of a broad distinction between experience and agency, our studies consistently revealed three fundamental components of mental life-suites of capacities related to the body, the heart, and the mind-with each component encompassing related aspects of both experience and agency. This body-heart-mind framework distinguishes itself from Gray et al.'s experience-agency framework by its clear and importantly different implications for dehumanization, moral reasoning, and other important social phenomena.

Infants possess an abstract expectation of ingroup support.

One pervasive facet of human interactions is the tendency to favor ingroups over outgroups. Remarkably, this tendency has been observed even when individuals are assigned to minimal groups based on arbitrary markers. Why is mere categorization into a minimal group sufficient to elicit some degree of ingroup favoritism? We consider several accounts that have been proposed in answer to this question and then test one particular account, which holds that ingroup favoritism reflects in part an abstract and early-emerging sociomoral expectation of ingroup support. In violation-of-expectation experiments with 17-mo-old infants, unfamiliar women were first identified (using novel labels) as belonging to the same group, to different groups, or to unspecified groups. Next, one woman needed instrumental assistance to achieve her goal, and another woman either provided the necessary assistance (help event) or chose not to do so (ignore event). When the two women belonged to the same group, infants looked significantly longer if shown the ignore as opposed to the help event; when the two women belonged to different groups or to unspecified groups, however, infants looked equally at the two events. Together, these results indicate that infants view helping as expected among individuals from the same group, but as optional otherwise. As such, the results demonstrate that from an early age, an abstract expectation of ingroup support contributes to ingroup favoritism in human interactions.

Numerosity representation is encoded in human subcortex.

Certain numerical abilities appear to be relatively ubiquitous in the animal kingdom, including the ability to recognize and differentiate relative quantities. This skill is present in human adults and children, as well as in nonhuman primates and, perhaps surprisingly, is also demonstrated by lower species such as mosquitofish and spiders, despite the absence of cortical computation available to primates. This ubiquity of numerical competence suggests that representations that connect to numerical tasks are likely subserved by evolutionarily conserved regions of the nervous system. Here, we test the hypothesis that the evaluation of relative numerical quantities is subserved by lower-order brain structures in humans. Using a monocular/dichoptic paradigm, across four experiments, we show that the discrimination of displays, consisting of both large (5-80) and small (1-4) numbers of dots, is facilitated in the monocular, subcortical portions of the visual system. This is only the case, however, when observers evaluate larger ratios of 3:1 or 4:1, but not smaller ratios, closer to 1:1. This profile of competence matches closely the skill with which newborn infants and other species can discriminate numerical quantity. These findings suggest conservation of ontogenetically and phylogenetically lower-order systems in adults' numerical abilities. The involvement of subcortical structures in representing numerical quantities provokes a reconsideration of current theories of the neural basis of numerical cognition, inasmuch as it bolsters the cross-species continuity of the biological system for numerical abilities.

Perceiving social interactions in the posterior superior temporal sulcus.

Primates are highly attuned not just to social characteristics of individual agents, but also to social interactions between multiple agents. Here we report a neural correlate of the representation of social interactions in the human brain. Specifically, we observe a strong univariate response in the posterior superior temporal sulcus (pSTS) to stimuli depicting social interactions between two agents, compared with (i) pairs of agents not interacting with each other, (ii) physical interactions between inanimate objects, and (iii) individual animate agents pursuing goals and interacting with inanimate objects. We further show that this region contains information about the nature of the social interaction-specifically, whether one agent is helping or hindering the other. This sensitivity to social interactions is strongest in a specific subregion of the pSTS but extends to a lesser extent into nearby regions previously implicated in theory of mind and dynamic face perception. This sensitivity to the presence and nature of social interactions is not easily explainable in terms of low-level visual features, attention, or the animacy, actions, or goals of individual agents. This region may underlie our ability to understand the structure of our social world and navigate within it.