Deep social neuroscience: the promise and peril of using artificial neural networks to study the social brain.

This review offers an accessible primer to social neuroscientists interested in neural networks. It begins by providing an overview of key concepts in deep learning. It then discusses three ways neural networks can be useful to social neuroscientists: (i) building statistical models to predict behavior from brain activity; (ii) quantifying naturalistic stimuli and social interactions; and (iii) generating cognitive models of social brain function. These applications have the potential to enhance the clinical value of neuroimaging and improve the generalizability of social neuroscience research. We also discuss the significant practical challenges, theoretical limitations and ethical issues faced by deep learning. If the field can successfully navigate these hazards, we believe that artificial neural networks may prove indispensable for the next stage of the field's development: deep social neuroscience.

Neural representations of situations and mental states are composed of sums of representations of the actions they afford.

Human behavior depends on both internal and external factors. Internally, people's mental states motivate and govern their behavior. Externally, one's situation constrains which actions are appropriate or possible. To predict others' behavior, one must understand the influences of mental states and situations on actions. On this basis, we hypothesize that people represent situations and states in terms of associated actions. To test this, we use functional neuroimaging to estimate neural activity patterns associated with situations, mental states, and actions. We compute sums of the action patterns, weighted by how often each action occurs in each situation and state. We find that these summed action patterns reconstructed the corresponding situation and state patterns. These results suggest that neural representations of situations and mental states are composed of sums of their action affordances. Summed action representations thus offer a biological mechanism by which people can predict actions given internal and external factors.

The Emerging Science of Interacting Minds.

For over a century, psychology has focused on uncovering mental processes of a single individual. However, humans rarely navigate the world in isolation. The most important determinants of successful development, mental health, and our individual traits and preferences arise from interacting with other individuals. Social interaction underpins who we are, how we think, and how we behave. Here we discuss the key methodological challenges that have limited progress in establishing a robust science of how minds interact and the new tools that are beginning to overcome these challenges. A deep understanding of the human mind requires studying the context within which it originates and exists: social interaction.

Advancing Naturalistic Affective Science with Deep Learning.

People express their own emotions and perceive others' emotions via a variety of channels, including facial movements, body gestures, vocal prosody, and language. Studying these channels of affective behavior offers insight into both the experience and perception of emotion. Prior research has predominantly focused on studying individual channels of affective behavior in isolation using tightly controlled, non-naturalistic experiments. This approach limits our understanding of emotion in more naturalistic contexts where different channels of information tend to interact. Traditional methods struggle to address this limitation: manually annotating behavior is time-consuming, making it infeasible to do at large scale; manually selecting and manipulating stimuli based on hypotheses may neglect unanticipated features, potentially generating biased conclusions; and common linear modeling approaches cannot fully capture the complex, nonlinear, and interactive nature of real-life affective processes. In this methodology review, we describe how deep learning can be applied to address these challenges to advance a more naturalistic affective science. First, we describe current practices in affective research and explain why existing methods face challenges in revealing a more naturalistic understanding of emotion. Second, we introduce deep learning approaches and explain how they can be applied to tackle three main challenges: quantifying naturalistic behaviors, selecting and manipulating naturalistic stimuli, and modeling naturalistic affective processes. Finally, we describe the limitations of these deep learning methods, and how these limitations might be avoided or mitigated. By detailing the promise and the peril of deep learning, this review aims to pave the way for a more naturalistic affective science.

Transition dynamics shape mental state concepts.

People have a unique ability to represent other people's internal thoughts and feelings-their mental states. Mental state knowledge has a rich conceptual structure, organized along key dimensions, such as valence. People use this conceptual structure to guide social interactions. How do people acquire their understanding of this structure? Here we investigate an underexplored contributor to this process: observation of mental state dynamics. Mental states-including both emotions and cognitive states-are not static. Rather, the transitions from one state to another are systematic and predictable. Drawing on prior cognitive science, we hypothesize that these transition dynamics may shape the conceptual structure that people learn to apply to mental states. Across nine behavioral experiments (N = 1,439), we tested whether the transition probabilities between mental states causally shape people's conceptual judgments of those states. In each study, we found that observing frequent transitions between mental states caused people to judge them to be conceptually similar. Computational modeling indicated that people translated mental state dynamics into concepts by embedding the states as points within a geometric space. The closer two states are within this space, the greater the likelihood of transitions between them. In three neural network experiments, we trained artificial neural networks to predict real human mental state dynamics. The networks spontaneously learned the same conceptual dimensions that people use to understand mental states. Together these results indicate that mental state dynamics-and the goal of predicting them-shape the structure of mental state concepts. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The 3d Mind Model characterizes how people understand mental states across modern and historical cultures.

Humans rely on social interaction to achieve many important goals. These interactions rely in turn on people's capacity to understand others' mental states: their thoughts and feelings. Do different cultures understand minds in different ways, or do widely shared principles describe how different cultures understand mental states? Extensive data suggest that the mind organizes mental state concepts using the 3d Mind Model, composed of the psychological dimensions: rationality (vs. emotionality), social impact (states which affect others more vs. less), and valence (positive vs. negative states). However, this evidence comes primarily from English-speaking individuals in the United States. Here we investigated mental state representation in 57 contemporary countries, using 163 million English language tweets; in 17 languages, using billions of words of text from internet webpages; and across more than 2000 years of history, using curated texts from four historical societies. We quantified mental state meaning by analyzing the text produced by each culture using word embeddings. We then tested whether the 3d Mind Model could explain which mental states were similar in meaning within each culture. We found that the 3d Mind Model significantly explained mental state meaning in every country, language, and historical society that we examined. These results suggest that rationality, social impact, and valence form a generalizable conceptual backbone for mental state representation.

Accurate emotion prediction in dyads and groups and its potential social benefits.

Emotion dynamics vary considerably from individual to individual and from group to group. Successful social interactions require people to track this moving target in order to anticipate the thoughts, feelings, and actions of others. In two studies, we test whether people track others' emotional idiosyncrasies to make accurate, target-specific emotion predictions. In both studies, participants predicted the emotion transitions of a specific target-either a close friend (Study 1) or a first-year college roommate (Study 2)-as well as an average group member. Results demonstrate that people can make highly accurate predictions both for specific individuals and specific groups. Accurate predictions rely on target-specific knowledge; new community members were able to make accurate predictions at zero-acquaintance, but accuracy increased over time as individuals accrued specialized knowledge. Results also suggest that accurate emotion prediction is associated with social success in both individual and communal relationships and that such a relation might emerge over time. Overall, our studies suggest that people accurately make individualized predictions of others' emotion transitions and that doing so fulfills a meaningful function in the social world. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Six dimensions describe action understanding: The ACT-FASTaxonomy.

Humans engage in a wide variety of different actions and activities. These range from simple motor actions like reaching for an object, to complex activities like governing a nation. Navigating everyday life requires people to make sense of this diversity of actions. We suggest that the mind simplifies this complex domain by attending primarily to the most essential features of actions. Using a parsimonious set of action dimensions, the mind can organize action knowledge in a low-dimensional representational space. In seven studies, we derive and validate such an action taxonomy. Study 1 uses large-scale text analyses to generate and test potential action dimensions. Study 2 validates interpretable labels for these dimensions. Studies 3-5 demonstrate that these dimensions can explain human judgments about actions. We perform model selection on data from these studies to arrive at the optimal set of six psychological dimensions, together forming the Abstraction, Creation, Tradition, Food, Animacy, Spiritualism Taxonomy (ACT-FAST). Study 6 demonstrates that ACT-FAST can predict socially relevant qualities of actions, including how, when, where, why, and by whom they are performed. Finally, Study 7 shows that ACT-FAST can explain action-related patterns of brain activity using naturalistic functional MRI (MRI). Together, these studies reveal the dimensional structure the mind applies to organize action concepts. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Inferring Mycobacterium bovis transmission between cattle and badgers using isolates from the Randomised Badger Culling Trial.

Mycobacterium bovis (M. bovis) is a causative agent of bovine tuberculosis, a significant source of morbidity and mortality in the global cattle industry. The Randomised Badger Culling Trial was a field experiment carried out between 1998 and 2005 in the South West of England. As part of this trial, M. bovis isolates were collected from contemporaneous and overlapping populations of badgers and cattle within ten defined trial areas. We combined whole genome sequences from 1,442 isolates with location and cattle movement data, identifying transmission clusters and inferred rates and routes of transmission of M. bovis. Most trial areas contained a single transmission cluster that had been established shortly before sampling, often contemporaneous with the expansion of bovine tuberculosis in the 1980s. The estimated rate of transmission from badger to cattle was approximately two times higher than from cattle to badger, and the rate of within-species transmission considerably exceeded these for both species. We identified long distance transmission events linked to cattle movement, recurrence of herd breakdown by infection within the same transmission clusters and superspreader events driven by cattle but not badgers. Overall, our data suggests that the transmission clusters in different parts of South West England that are still evident today were established by long-distance seeding events involving cattle movement, not by recrudescence from a long-established wildlife reservoir. Clusters are maintained primarily by within-species transmission, with less frequent spill-over both from badger to cattle and cattle to badger.

Mask exposure during COVID-19 changes emotional face processing.

Faces are one of the key ways that we obtain social information about others. They allow people to identify individuals, understand conversational cues, and make judgements about others' mental states. When the COVID-19 pandemic hit the United States, widespread mask-wearing practices were implemented, causing a shift in the way Americans typically interact. This introduction of masks into social exchanges posed a potential challenge-how would people make these important inferences about others when a large source of information was no longer available? We conducted two studies that investigated the impact of mask exposure on emotion perception. In particular, we measured how participants used facial landmarks (visual cues) and the expressed valence and arousal (affective cues), to make similarity judgements about pairs of emotion faces. Study 1 found that in August 2020, participants with higher levels of mask exposure used cues from the eyes to a greater extent when judging emotion similarity than participants with less mask exposure. Study 2 measured participants' emotion perception in both April and September 2020 -before and after widespread mask adoption-in the same group of participants to examine changes in the use of facial cues over time. Results revealed an overall increase in the use of visual cues from April to September. Further, as mask exposure increased, people with the most social interaction showed the largest increase in the use of visual facial cues. These results provide evidence that a shift has occurred in how people process faces such that the more people are interacting with others that are wearing masks, the more they have learned to focus on visual cues from the eye area of the face.

The representational structure of mental states generalizes across target people and stimulus modalities.

Each individual experiences mental states in their own idiosyncratic way, yet perceivers can accurately understand a huge variety of states across unique individuals. How do they accomplish this feat? Do people think about their own anger in the same ways as another person's anger? Is reading about someone's anxiety the same as seeing it? Here, we test the hypothesis that a common conceptual core unites mental state representations across contexts. Across three studies, participants judged the mental states of multiple targets, including a generic other, the self, a socially close other, and a socially distant other. Participants viewed mental state stimuli in multiple modalities, including written scenarios and images. Using representational similarity analysis, we found that brain regions associated with social cognition expressed stable neural representations of mental states across both targets and modalities. Together, these results suggest that people use stable models of mental states across different people and contexts.

People accurately predict the transition probabilities between actions.

Social life is a complex dance. To coordinate gracefully with one's partners, one must predict their actions. Here, we investigated how people predict others' actions. We hypothesized that people can accurately predict others' future actions based on knowledge of their current actions, coupled with knowledge of action transitions. To test whether people have accurate knowledge of the transition probabilities between actions, we compared actual rates of action transitions-calculated from four large naturalistic datasets-to participants' ratings of the transition probabilities between corresponding sets of actions. In five preregistered studies, participants demonstrated accurate mental models of action transitions. Furthermore, we found that people drew upon conceptual knowledge of actions-described by the six-dimensional ACT-FASTaxonomy-to guide their accurate predictions. Together, these results indicate that people can accurately anticipate other people's moves in the dance of social life and that the structure of action knowledge may be tailored to making these predictions.

Real-world outcomes of patients with locally advanced or metastatic epithelioid sarcoma.

BACKGROUND: Limited data are available on the real-world effectiveness and safety of systemic therapies for advanced (surgically unresectable and/or metastatic) epithelioid sarcoma (ES). METHODS: A retrospective medical records review was conducted in patients with advanced ES who were initiating first-line or ≥2 lines of systemic therapy (2000-2017) at 5 US cancer centers. The real-world overall response rate (rwORR), the duration of response (rwDOR), the disease control rate (rwDCR) (defined as stable disease for ≥32 weeks or any duration of response), and progression-free survival (rwPFS) were assessed by radiology reports. Overall survival (OS), rwDOR, and rwPFS were estimated from the time therapy was initiated using the Kaplan-Meier method. Serious adverse events were assessed. RESULTS: Of 74 patients (median age at diagnosis, 33 years; range, 10.6-76.3 years), 72% were male, and 85% had metastatic disease. The median number of lines of therapy was 2 (range, 1-7 lines of therapy), and 46 patients (62%) received ≥2 lines of systemic therapy. First-line regimens were usually anthracycline-based (54%) or gemcitabine-based (24%). For patients receiving first-line systemic therapy, the rwORR was 15%, the rwDCR was 20%, the median rwDOR was 3.3 months (95% CI, 2.1-5.2 months), the median rwPFS was 2.5 months (95% CI, 1.7, 6.9 months), and the median OS was 15.2 months (95% CI, 11.4-21.7 months). For those who received ≥2 lines of systemic therapy, the rwORR was 9%, the rwDCR was 20%, the median rwDOR was 4.5 months (95% CI, 0.7-5.6 months), and the median rwPFS was 6.0 months (95% CI, 3.2-7.4 months). Over one-half of patients (51.4%) experienced an adverse event, most frequently febrile neutropenia (14%), pain (10%), anemia, dyspnea, fever, thrombocytopenia, or transaminitis (5% each). CONCLUSIONS: Systemic therapies demonstrate limited efficacy in patients with advanced ES and have associated toxicities.

Perceiving actions before they happen: psychological dimensions scaffold neural action prediction.

The social world buzzes with action. People constantly walk, talk, eat, work, play, snooze and so on. To interact with others successfully, we need to both understand their current actions and predict their future actions. Here we used functional neuroimaging to test the hypothesis that people do both at the same time: when the brain perceives an action, it simultaneously encodes likely future actions. Specifically, we hypothesized that the brain represents perceived actions using a map that encodes which actions will occur next: the six-dimensional Abstraction, Creation, Tradition, Food(-relevance), Animacy and Spiritualism Taxonomy (ACT-FAST) action space. Within this space, the closer two actions are, the more likely they are to precede or follow each other. To test this hypothesis, participants watched a video featuring naturalistic sequences of actions while undergoing functional magnetic resonance imaging (fMRI) scanning. We first use a decoding model to demonstrate that the brain uses ACT-FAST to represent current actions. We then successfully predicted as-yet unseen actions, up to three actions into the future, based on their proximity to the current action's coordinates in ACT-FAST space. This finding suggests that the brain represents actions using a six-dimensional action space that gives people an automatic glimpse of future actions.

People represent mental states in terms of rationality, social impact, and valence: Validating the 3d Mind Model.

Humans can experience a wide variety of different thoughts and feelings in the course of everyday life. To successfully navigate the social world, people need to perceive, understand, and predict others' mental states. Previous research suggests that people use three dimensions to represent mental states: rationality, social impact, and valence. This 3d Mind Model allows people to efficiently "see" the state of another person's mind by considering whether that state is rational or emotional, more or less socially impactful, and positive or negative. In the current investigation, we validate this model using neural, behavioral, and linguistic evidence. First, we examine the robustness of the 3d Mind Model by conducting a mega-analysis of four fMRI studies in which participants considered others' mental states. We find evidence that rationality, social impact, and valence each contribute to explaining the neural representation of mental states. Second, we test whether the 3d Mind Model offers the optimal combination of dimensions for describing neural representations of mental state. Results reveal that the 3d Mind Model achieve the best performance among a large set of candidate dimensions. Indeed, it offers a highly explanatory account of mental state representation, explaining over 80% of reliable neural variance. Finally, we demonstrate that all three dimensions of the model likewise capture convergent behavioral and linguistic measures of mental state representation. Together, these findings provide strong support for the 3d Mind Model, indicating that is it is a robust and generalizable account of how people think about mental states.

The brain represents people as the mental states they habitually experience.

Social life requires us to treat each person according to their unique disposition. To tailor our behavior to specific individuals, we must represent their idiosyncrasies. Here, we advance the hypothesis that our representations of other people reflect the mental states we perceive those people to habitually experience. We tested this hypothesis by measuring whether neural representations of people could be accurately reconstructed by summing state representations. Separate participants underwent functional MRI while considering famous individuals and individual mental states. Online participants rated how often each famous person experiences each state. Results supported the summed state hypothesis: frequency-weighted sums of state-specific brain activity patterns accurately reconstructed person-specific patterns. Moreover, the summed state account outperformed the established alternative-that people represent others using trait dimensions-in explaining interpersonal similarity. These findings demonstrate that the brain represents people as the sums of the mental states they experience.

People represent their own mental states more distinctly than those of others.

One can never know the internal workings of another person-one can only infer others' mental states based on external cues. In contrast, each person has direct access to the contents of their own mind. Here, we test the hypothesis that this privileged access shapes the way people represent internal mental experiences, such that they represent their own mental states more distinctly than the states of others. Across four studies, participants considered their own and others' mental states; analyses measured the distinctiveness of mental state representations. Two fMRI studies used representational similarity analyses to demonstrate that the social brain manifests more distinct activity patterns when thinking about one's own states vs. others'. Two behavioral studies complement these findings, and demonstrate that people differentiate between states less as social distance increases. Together, these results suggest that we represent our own mind with greater granularity than the minds of others.

The Social Brain Automatically Predicts Others' Future Mental States.

Social life requires people to predict the future: people must anticipate others' thoughts, feelings, and actions to interact with them successfully. The theory of predictive coding suggests that the social brain may meet this need by automatically predicting others' social futures. If so, when representing others' current mental state, the brain should already start representing their future states. To test this hypothesis, we used fMRI to measure female and male human participants' neural representations of mental states. Representational similarity analysis revealed that neural patterns associated with mental states currently under consideration resembled patterns of likely future states more so than patterns of unlikely future states. This effect manifested in activity across the social brain network and in medial prefrontal cortex in particular. Repetition suppression analysis also supported the social predictive coding hypothesis: considering mental states presented in predictable sequences reduced activity in the precuneus relative to unpredictable sequences. In addition to demonstrating that the brain makes automatic predictions of others' social futures, the results also demonstrate that the brain leverages a 3D representational space to make these predictions. Proximity between mental states on the psychological dimensions of rationality, social impact, and valence explained much of the association between state-specific neural pattern similarity and state transition likelihood. Together, these findings suggest that the way the brain represents the social present gives people an automatic glimpse of the social future.SIGNIFICANCE STATEMENT When you see a ball in flight, your brain calculates, not just its static visual features such as size and shape, but also predicts its future trajectory. Here, we investigated whether the same might hold true in the social world: when we see someone flying into a rage, does our brain automatically predict their social trajectory? In this study, we scanned participants' brain activity while they judged others' mental states. We found that neural activity associated with a given state resembled activity associated with likely future states. Additionally, unpredictable sequences of states evoked more brain activity than predictable sequences, consistent with monitoring for, and updating from, prediction errors. These results suggest that the social brain automatically predicts others' future mental states.

Modeling the Predictive Social Mind.

The social mind is tailored to the problem of predicting the mental states and actions of other people. However, social cognition researchers have only scratched the surface of the predictive social mind. We discuss here a new framework for explaining how people organize social knowledge and use it for social prediction. Specifically, we propose a multilayered framework of social cognition in which two hidden layers - the mental states and traits of others - support predictions about the observable layer - the actions of others. A parsimonious set of psychological dimensions structures each layer, and proximity within and across layers guides social prediction. This simple framework formalizes longstanding intuitions from social cognition, and in doing so offers a generative model for deriving new hypotheses about predictive social cognition.

Theories of Person Perception Predict Patterns of Neural Activity During Mentalizing.

Social life requires making inferences about other people. What information do perceivers spontaneously draw upon to make such inferences? Here, we test 4 major theories of person perception, and 1 synthetic theory that combines their features, to determine whether the dimensions of such theories can serve as bases for describing patterns of neural activity during mentalizing. While undergoing functional magnetic resonance imaging, participants made social judgments about well-known public figures. Patterns of brain activity were then predicted using feature encoding models that represented target people's positions on theoretical dimensions such as warmth and competence. All 5 theories of person perception proved highly accurate at reconstructing activity patterns, indicating that each could describe the informational basis of mentalizing. Cross-validation indicated that the theories robustly generalized across both targets and participants. The synthetic theory consistently attained the best performance-approximately two-thirds of noise ceiling accuracy--indicating that, in combination, the theories considered here can account for much of the neural representation of other people. Moreover, encoding models trained on the present data could reconstruct patterns of activity associated with mental state representations in independent data, suggesting the use of a common neural code to represent others' traits and states.