Smartphone-based gaze estimation for in-home autism research.

Atypical gaze patterns are a promising biomarker of autism spectrum disorder. To measure gaze accurately, however, it typically requires highly controlled studies in the laboratory using specialized equipment that is often expensive, thereby limiting the scalability of these approaches. Here we test whether a recently developed smartphone-based gaze estimation method could overcome such limitations and take advantage of the ubiquity of smartphones. As a proof-of-principle, we measured gaze while a small sample of well-assessed autistic participants and controls watched videos on a smartphone, both in the laboratory (with lab personnel) and in remote home settings (alone). We demonstrate that gaze data can be efficiently collected, in-home and longitudinally by participants themselves, with sufficiently high accuracy (gaze estimation error below 1° visual angle on average) for quantitative, feature-based analysis. Using this approach, we show that autistic individuals have reduced gaze time on human faces and longer gaze time on non-social features in the background, thereby reproducing established findings in autism using just smartphones and no additional hardware. Our approach provides a foundation for scaling future research with larger and more representative participant groups at vastly reduced cost, also enabling better inclusion of underserved communities.

Commentary on, "The Human Affectome," by Schiller et al.

I suggest that this project could benefit from a relational database of some sort to provide readers with a more formal ontology, and that the authors consider making a distinction between experiential and functional aspects of emotion.

Opinion: Which animals have personality?

Human personality generally refers to coherent individuating patterns in affect, behavior, and cognition. We can only observe and measure behavior, from which we then infer personality and other psychological processes (affect, cognition, etc.). We emphasize that the study of personality always explains or summarizes patterns not only in behavior but also in these other psychological processes inferred from behavior. We thus argue that personality should be attributed only to nonhuman animals with behaviors from which we can infer a sufficiently rich set of psychological processes. The mere inference of a biological trait that explains behavioral variability, on our view, is not sufficient to count as a personality construct and should be given a different term. Methodologically, inferring personality in nonhuman animals entails challenges in characterizing ecologically valid behaviors, doing so across rich and varied environments, and collecting enough data. We suggest that studies should gradually accumulate such corpora of data on a species through well-curated shared databases. A mixture of approaches should include both top-down fit with extant human personality theories (such as the Big Five) as well as bottom-up discovery of species-specific personality dimensions. Adopting the above framework will help us to build a comparative psychology and will provide the most informative models also for understanding human personality, its evolution, and its disorders.

Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients.

We present a multimodal dataset of intracranial recordings, fMRI, and eye tracking in 20 participants during movie watching. Recordings consist of single neurons, local field potential, and intracranial EEG activity acquired from depth electrodes targeting the amygdala, hippocampus, and medial frontal cortex implanted for monitoring of epileptic seizures. Participants watched an 8-min long excerpt from the video "Bang! You're Dead" and performed a recognition memory test for movie content. 3 T fMRI activity was recorded prior to surgery in 11 of these participants while performing the same task. This NWB- and BIDS-formatted dataset includes spike times, field potential activity, behavior, eye tracking, electrode locations, demographics, and functional and structural MRI scans. For technical validation, we provide signal quality metrics, assess eye tracking quality, behavior, the tuning of cells and high-frequency broadband power field potentials to familiarity and event boundaries, and show brain-wide inter-subject correlations for fMRI. This dataset will facilitate the investigation of brain activity during movie watching, recognition memory, and the neural basis of the fMRI-BOLD signal.

The Future of Affective Science: Introduction to the Special Issue.

Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.

Collective events and individual affect shape autobiographical memory.

How do collective events shape how we remember our lives? We leveraged advances in natural language processing as well as a rich, longitudinal assessment of 1,000 Americans throughout 2020 to examine how memory is influenced by two prominent factors: surprise and emotion. Autobiographical memory for 2020 displayed a unique signature: There was a substantial bump in March, aligning with pandemic onset and lockdowns, consistent across three memory collections 1 y apart. We further investigated how emotion, using both immediate and retrieved measures, predicted the amount and content of autobiographical memory: Negative affect increased recall across all measures, whereas its more clinical indices, depression and posttraumatic stress disorder, selectively increased nonepisodic recall. Finally, in a separate cohort, we found pandemic news to be better remembered, surprising, and negative, while lockdowns compressed remembered time. Our work connects laboratory findings to the real world and delineates the effects of acute versus clinical signatures of negative emotion on memory.

Neural signatures of social inferences predict the number of real-life social contacts and autism severity.

We regularly infer other people's thoughts and feelings from observing their actions, but how this ability contributes to successful social behavior and interactions remains unknown. We show that neural activation patterns during social inferences obtained in the laboratory predict the number of social contacts in the real world, as measured by the social network index, in three neurotypical samples (total n = 126) and one sample of autistic adults (n = 23). We also show that brain patterns during social inference generalize across individuals in these groups. Cross-validated associations between brain activations and social inference localize selectively to the right posterior superior temporal sulcus and were specific for social, but not nonsocial, inference. Activation within this same brain region also predicts autism-like trait scores from questionnaires and autism symptom severity. Thus, neural activations produced while thinking about other people's mental states predict variance in multiple indices of social functioning in the real world.

Dissociable brain structural asymmetry patterns reveal unique phenome-wide profiles.

Broca reported ~150 years ago that particular lesions of the left hemisphere impair speech. Since then, other brain regions have been reported to show lateralized structure and function. Yet, studies of brain asymmetry have limited their focus to pairwise comparisons between homologous regions. Here, we characterized separable whole-brain asymmetry patterns in grey and white matter structure from n = 37,441 UK Biobank participants. By pooling information on left-right shifts underlying whole-brain structure, we deconvolved signatures of brain asymmetry that are spatially distributed rather than locally constrained. Classically asymmetric regions turned out to belong to more than one asymmetry pattern. Instead of a single dominant signature, we discovered complementary asymmetry patterns that contributed similarly to whole-brain asymmetry at the population level. These asymmetry patterns were associated with unique collections of phenotypes, ranging from early lifestyle factors to demographic status to mental health indicators.

Trait impressions from faces depend on the goals of the perceiver.

Trait impressions from faces formed in the real world likely depend on the circumstances in which a face is seen, in particular, on the goal of the perceiver in that circumstance. This goal dependency is typically not incorporated into laboratory studies, an omission that has limited our understanding of trait impressions from faces.

Atypical gaze patterns in autistic adults are heterogeneous across but reliable within individuals.

BACKGROUND: Across behavioral studies, autistic individuals show greater variability than typically developing individuals. However, it remains unknown to what extent this variability arises from heterogeneity across individuals, or from unreliability within individuals. Here, we focus on eye tracking, which provides rich dependent measures that have been used extensively in studies of autism. Autistic individuals have an atypical gaze onto both static visual images and dynamic videos that could be leveraged for diagnostic purposes if the above open question could be addressed. METHODS: We tested three competing hypotheses: (1) that gaze patterns of autistic individuals are less reliable or noisier than those of controls, (2) that atypical gaze patterns are individually reliable but heterogeneous across autistic individuals, or (3) that atypical gaze patterns are individually reliable and also homogeneous among autistic individuals. We collected desktop-based eye tracking data from two different full-length television sitcom episodes, at two independent sites (Caltech and Indiana University), in a total of over 150 adult participants (N = 48 autistic individuals with IQ in the normal range, 105 controls) and quantified gaze onto features of the videos using automated computer vision-based feature extraction. RESULTS: We found support for the second of these hypotheses. Autistic people and controls showed equivalently reliable gaze onto specific features of videos, such as faces, so much so that individuals could be identified significantly above chance using a fingerprinting approach from video epochs as short as 2 min. However, classification of participants into diagnostic groups based on their eye tracking data failed to produce clear group classifications, due to heterogeneity in the autistic group. LIMITATIONS: Three limitations are the relatively small sample size, assessment across only two videos (from the same television series), and the absence of other dependent measures (e.g., neuroimaging or genetics) that might have revealed individual-level variability that was not evident with eye tracking. Future studies should expand to larger samples across longer longitudinal epochs, an aim that is now becoming feasible with Internet- and phone-based eye tracking. CONCLUSIONS: These findings pave the way for the investigation of autism subtypes, and for elucidating the specific visual features that best discriminate gaze patterns-directions that will also combine with and inform neuroimaging and genetic studies of this complex disorder.

Mapping effective connectivity of human amygdala subdivisions with intracranial stimulation.

The primate amygdala is a complex consisting of over a dozen nuclei that have been implicated in a host of cognitive functions, individual differences, and psychiatric illnesses. These functions are implemented through distinct connectivity profiles, which have been documented in animals but remain largely unknown in humans. Here we present results from 25 neurosurgical patients who had concurrent electrical stimulation of the amygdala with intracranial electroencephalography (electrical stimulation tract-tracing; es-TT), or fMRI (electrical stimulation fMRI; es-fMRI), methods providing strong inferences about effective connectivity of amygdala subdivisions with the rest of the brain. We quantified functional connectivity with medial and lateral amygdala, the temporal order of these connections on the timescale of milliseconds, and also detail second-order effective connectivity among the key nodes. These findings provide a uniquely detailed characterization of human amygdala functional connectivity that will inform functional neuroimaging studies in healthy and clinical populations.

The geometry of domain-general performance monitoring in the human medial frontal cortex.

Controlling behavior to flexibly achieve desired goals depends on the ability to monitor one's own performance. It is unknown how performance monitoring can be both flexible, to support different tasks, and specialized, to perform each task well. We recorded single neurons in the human medial frontal cortex while subjects performed two tasks that involve three types of cognitive conflict. Neurons encoding conflict probability, conflict, and error in one or both tasks were intermixed, forming a representational geometry that simultaneously allowed task specialization and generalization. Neurons encoding conflict retrospectively served to update internal estimates of conflict probability. Population representations of conflict were compositional. These findings reveal how representations of evaluative signals can be both abstract and task-specific and suggest a neuronal mechanism for estimating control demand.

Caltech Conte Center, a multimodal data resource for exploring social cognition and decision-making.

This data release of 117 healthy community-dwelling adults provides multimodal high-quality neuroimaging and behavioral data for the investigation of brain-behavior relationships. We provide structural MRI, resting-state functional MRI, movie functional MRI, together with questionnaire-based and task-based psychological variables; many of the participants have multiple datasets from retesting over the course of several years. Our dataset is distinguished by utilizing open-source data formats and processing tools (BIDS, FreeSurfer, fMRIPrep, MRIQC), providing data that is thoroughly quality checked, preprocessed to various extents and available in multiple anatomical spaces. A customizable denoising pipeline is provided as open-source code that includes tools for the generation of functional connectivity matrices and initialization of individual difference analyses. Behavioral data include a comprehensive set of psychological assessments on gold-standard instruments encompassing cognitive function, mood and personality, together with exploratory factor analyses. The dataset provides an in-depth, multimodal resource for investigating associations between individual differences, brain structure and function, with a focus on the domains of social cognition and decision-making.

Video-evoked fMRI BOLD responses are highly consistent across different data acquisition sites.

Naturalistic imaging paradigms, in which participants view complex videos in the scanner, are increasingly used in human cognitive neuroscience. Videos evoke temporally synchronized brain responses that are similar across subjects as well as within subjects, but the reproducibility of these brain responses across different data acquisition sites has not yet been quantified. Here, we characterize the consistency of brain responses across independent samples of participants viewing the same videos in functional magnetic resonance imaging (fMRI) scanners at different sites (Indiana University and Caltech). We compared brain responses collected at these different sites for two carefully matched datasets with identical scanner models, acquisition, and preprocessing details, along with a third unmatched dataset in which these details varied. Our overall conclusion is that for matched and unmatched datasets alike, video-evoked brain responses have high consistency across these different sites, both when compared across groups and across pairs of individuals. As one might expect, differences between sites were larger for unmatched datasets than matched datasets. Residual differences between datasets could in part reflect participant-level variability rather than scanner- or data- related effects. Altogether our results indicate promise for the development and, critically, generalization of video fMRI studies of individual differences in healthy and clinical populations alike.

Ethical commitments, principles, and practices guiding intracranial neuroscientific research in humans.

Leveraging firsthand experience, BRAIN-funded investigators conducting intracranial human neuroscience research propose two fundamental ethical commitments: (1) maintaining the integrity of clinical care and (2) ensuring voluntariness. Principles, practices, and uncertainties related to these commitments are offered for future investigation.

A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks.

People spontaneously infer other people's psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s42761-021-00075-5.

Reorganization of the Social Brain in Individuals with Only One Intact Cerebral Hemisphere.

Social cognition and emotion are ubiquitous human processes that recruit a reliable set of brain networks in healthy individuals. These brain networks typically comprise midline (e.g., medial prefrontal cortex) as well as lateral regions of the brain including homotopic regions in both hemispheres (e.g., left and right temporo-parietal junction). Yet the necessary roles of these networks, and the broader roles of the left and right cerebral hemispheres in socioemotional functioning, remains debated. Here, we investigated these questions in four rare adults whose right (three cases) or left (one case) cerebral hemisphere had been surgically removed (to a large extent) to treat epilepsy. We studied four closely matched healthy comparison participants, and also compared the patient findings to data from a previously published larger healthy comparison sample (n = 33). Participants completed standardized socioemotional and cognitive assessments to investigate social cognition. Functional magnetic resonance imaging (fMRI) data were obtained during passive viewing of a short, animated movie that distinctively recruits two social brain networks: one engaged when thinking about other agents' internal mental states (e.g., beliefs, desires, emotions; so-called Theory of Mind or ToM network), and the second engaged when thinking about bodily states (e.g., pain, hunger; so-called PAIN network). Behavioral assessments demonstrated remarkably intact general cognitive functioning in all individuals with hemispherectomy. Social-emotional functioning was somewhat variable in the hemispherectomy participants, but strikingly, none of these individuals had consistently impaired social-emotional processing and none of the assessment scores were consistent with a psychiatric disorder. Using inter-region correlation analyses, we also found surprisingly typical ToM and PAIN networks, as well as typical differentiation of the two networks (in the intact hemisphere of patients with either right or left hemispherectomy), based on idiosyncratic reorganization of cortical activation. The findings argue that compensatory brain networks can process social and emotional information following hemispherectomy across different age levels (from 3 months to 20 years old), and suggest that social brain networks typically distributed across midline and lateral brain regions in this domain can be reorganized, to a substantial degree.

Four dimensions characterize attributions from faces using a representative set of English trait words.

People readily (but often inaccurately) attribute traits to others based on faces. While the details of attributions depend on the language available to describe social traits, psychological theories argue that two or three dimensions (such as valence and dominance) summarize social trait attributions from faces. However, prior work has used only a small number of trait words (12 to 18), limiting conclusions to date. In two large-scale, preregistered studies we ask participants to rate 100 faces (obtained from existing face stimuli sets), using a list of 100 English trait words that we derived using deep neural network analysis of words that have been used by other participants in prior studies to describe faces. In study 1 we find that these attributions are best described by four psychological dimensions, which we interpret as "warmth", "competence", "femininity", and "youth". In study 2 we partially reproduce these four dimensions using the same stimuli among additional participant raters from multiple regions around the world, in both aggregated and individual-level data. These results provide a comprehensive characterization of trait attributions from faces, although we note our conclusions are limited by the scope of our study (in particular we note only white faces and English trait words were included).