Clarifying status of DNNs as models of human vision.

On several key issues we agree with the commentators. Perhaps most importantly, everyone seems to agree that psychology has an important role to play in building better models of human vision, and (most) everyone agrees (including us) that deep neural networks (DNNs) will play an important role in modelling human vision going forward. But there are also disagreements about what models are for, how DNN-human correspondences should be evaluated, the value of alternative modelling approaches, and impact of marketing hype in the literature. In our view, these latter issues are contributing to many unjustified claims regarding DNN-human correspondences in vision and other domains of cognition. We explore all these issues in this response.

Successes and critical failures of neural networks in capturing human-like speech recognition.

Natural and artificial audition can in principle acquire different solutions to a given problem. The constraints of the task, however, can nudge the cognitive science and engineering of audition to qualitatively converge, suggesting that a closer mutual examination would potentially enrich artificial hearing systems and process models of the mind and brain. Speech recognition - an area ripe for such exploration - is inherently robust in humans to a number transformations at various spectrotemporal granularities. To what extent are these robustness profiles accounted for by high-performing neural network systems? We bring together experiments in speech recognition under a single synthesis framework to evaluate state-of-the-art neural networks as stimulus-computable, optimized observers. In a series of experiments, we (1) clarify how influential speech manipulations in the literature relate to each other and to natural speech, (2) show the granularities at which machines exhibit out-of-distribution robustness, reproducing classical perceptual phenomena in humans, (3) identify the specific conditions where model predictions of human performance differ, and (4) demonstrate a crucial failure of all artificial systems to perceptually recover where humans do, suggesting alternative directions for theory and model building. These findings encourage a tighter synergy between the cognitive science and engineering of audition.

A Computational Complexity Perspective on Segmentation as a Cognitive Subcomputation.

Computational feasibility is a widespread concern that guides the framing and modeling of natural and artificial intelligence. The specification of cognitive system capacities is often shaped by unexamined intuitive assumptions about the search space and complexity of a subcomputation. However, a mistaken intuition might make such initial conceptualizations misleading for what empirical questions appear relevant later on. We undertake here computational-level modeling and complexity analyses of segmentation - a widely hypothesized subcomputation that plays a requisite role in explanations of capacities across domains, such as speech recognition, music cognition, active sensing, event memory, action parsing, and statistical learning - as a case study to show how crucial it is to formally assess these assumptions. We mathematically prove two sets of results regarding computational hardness and search space size that may run counter to intuition, and position their implications with respect to existing views on the subcapacity.

Deep problems with neural network models of human vision.

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modeling approaches that focus on psychological data.

Dynamic neurocognitive changes in interoception after heart transplant.

Heart-brain integration dynamics are critical for interoception (i.e. the sensing of body signals). In this unprecedented longitudinal study, we assessed neurocognitive markers of interoception in patients who underwent orthotopic heart transplants and matched healthy controls. Patients were assessed longitudinally before surgery (T1), a few months later (T2) and a year after (T3). We assessed behavioural (heartbeat detection) and electrophysiological (heartbeat evoked potential) markers of interoception. Heartbeat detection task revealed that pre-surgery (T1) interoception was similar between patients and controls. However, patients were outperformed by controls after heart transplant (T2), but no such differences were observed in the follow-up analysis (T3). Neurophysiologically, although heartbeat evoked potential analyses revealed no differences between groups before the surgery (T1), reduced amplitudes of this event-related potential were found for the patients in the two post-transplant stages (T2, T3). All these significant effects persisted after covariation with different cardiological measures. In sum, this study brings new insights into the adaptive properties of brain-heart pathways.

Against the Epistemological Primacy of the Hardware: The Brain from Inside Out, Turned Upside Down.

Before he wrote the recent book The Brain from Inside Out, the neuroscientist György Buzsáki previewed some of the arguments in a paper written 20 years ago ("The brain-cognitive behavior problem: a retrospective"), now finally published. The principal focus of the paper is the relationship between neuroscience and psychology. The direction in which that research had proceeded, and continues now, is, in his view, fundamentally misguided. Building on the critique, Buzsáki presents arguments for an "inside-out" approach, wherein the study of neurobiological objects has primacy over using psychological concepts to study the brain, and should, in fact, give rise to them. We argue that he is too pessimistic, and actually not quite right, about how the relation between cognition and neuroscience can be studied. Second, we are not in agreement with the normative recommendation of how to proceed: a predominantly brain first, implementation-driven research agenda. Finally, we raise concerns about the philosophical underpinning of the research program he advances. Buzsáki's perspective merits careful examination, and we suggest that it can be linked in a productive way to ongoing research, aligning his inside-out approach with current work that yields convincing accounts of mind and brain.

Time to Face Language: Embodied Mechanisms Underpin the Inception of Face-Related Meanings in the Human Brain.

In construing meaning, the brain recruits multimodal (conceptual) systems and embodied (modality-specific) mechanisms. Yet, no consensus exists on how crucial the latter are for the inception of semantic distinctions. To address this issue, we combined electroencephalographic (EEG) and intracranial EEG (iEEG) to examine when nouns denoting facial body parts (FBPs) and nonFBPs are discriminated in face-processing and multimodal networks. First, FBP words increased N170 amplitude (a hallmark of early facial processing). Second, they triggered fast (~100 ms) activity boosts within the face-processing network, alongside later (~275 ms) effects in multimodal circuits. Third, iEEG recordings from face-processing hubs allowed decoding ~80% of items before 200 ms, while classification based on multimodal-network activity only surpassed ~70% after 250 ms. Finally, EEG and iEEG connectivity between both networks proved greater in early (0-200 ms) than later (200-400 ms) windows. Collectively, our findings indicate that, at least for some lexico-semantic categories, meaning is construed through fast reenactments of modality-specific experience.

Consistent Gradient of Performance and Decoding of Stimulus Type and Valence From Local and Network Activity.

The individual differences approach focuses on the variation of behavioral and neural signatures across subjects. In this context, we searched for intracranial neural markers of performance in three individuals with distinct behavioral patterns (efficient, borderline, and inefficient) in a dual-valence task assessing facial and lexical emotion recognition. First, we performed a preliminary study to replicate well-established evoked responses in relevant brain regions. Then, we examined time series data and network connectivity, combined with multivariate pattern analyses and machine learning, to explore electrophysiological differences in resting-state versus task-related activity across subjects. Next, using the same methodological approach, we assessed the neural decoding of performance for different dimensions of the task. The classification of time series data mirrored the behavioral gradient across subjects for stimulus type but not for valence. However, network-based measures reflected the subjects' hierarchical profiles for both stimulus types and valence. Therefore, this measure serves as a sensitive marker for capturing distributed processes such as emotional valence discrimination, which relies on an extended set of regions. Network measures combined with classification methods may offer useful insights to study single subjects and understand inter-individual performance variability. Promisingly, this approach could eventually be extrapolated to other neuroscientific techniques.

Reading Shakespearean tropes in a foreign tongue: Age of L2 acquisition modulates neural responses to functional shifts.

Functional shifts (FSs) - morphosyntactically marked words evoking coherent but novel meanings - are ubiquitous in English and, specially, in Shakespearean literature. While their neural signatures have been explored in native speakers, no study has targeted foreign-language users, let alone comparing early and late bilinguals. Here, we administered a validated FS paradigm to subjects from both populations and evaluated time-frequency modulations evoked by FS and control sentences. Early bilinguals exhibited greater sensitivity towards FSs, indexed by reduced fronto-posterior theta-band oscillations across semantic- and structural-integration windows. Such oscillatory modulations may represent a key marker of age-of-acquisition effects during foreign-language wordplay processing.

Altered neural signatures of interoception in multiple sclerosis.

Multiple sclerosis (MS) patients present several alterations related to sensing of bodily signals. However, no specific neurocognitive impairment has yet been proposed as a core deficit underlying such symptoms. We aimed to determine whether MS patients present changes in interoception-that is, the monitoring of autonomic bodily information-a process that might be related to various bodily dysfunctions. We performed two studies in 34 relapsing-remitting, early-stage MS patients and 46 controls matched for gender, age, and education. In Study 1, we evaluated the heartbeat-evoked potential (HEP), a cortical signature of interoception, via a 128-channel EEG system during a heartbeat detection task including an exteroceptive and an interoceptive condition. Then, we obtained whole-brain MRI recordings. In Study 2, participants underwent fMRI recordings during two resting-state conditions: mind wandering and interoception. In Study 1, controls exhibited greater HEP modulation during the interoceptive condition than the exteroceptive one, but no systematic differences between conditions emerged in MS patients. Patients presented atrophy in the left insula, the posterior part of the right insula, and the right anterior cingulate cortex, with abnormal associations between neurophysiological and neuroanatomical patterns. In Study 2, controls showed higher functional connectivity and degree for the interoceptive state compared with mind wandering; however, this pattern was absent in patients, who nonetheless presented greater connectivity and degree than controls during mind wandering. MS patients were characterized by atypical multimodal brain signatures of interoception. This finding opens a new agenda to examine the role of inner-signal monitoring in the body symptomatology of MS.

Social neuroscience: undoing the schism between neurology and psychiatry.

Multiple disorders once jointly conceived as "nervous diseases" became segregated by the distinct institutional traditions forged in neurology and psychiatry. As a result, each field specialized in the study and treatment of a subset of such conditions. Here we propose new avenues for interdisciplinary interaction through a triangulation of both fields with social neuroscience. To this end, we review evidence from five relevant domains (facial emotion recognition, empathy, theory of mind, moral cognition, and social context assessment), highlighting their common disturbances across neurological and psychiatric conditions and discussing their multiple pathophysiological mechanisms. Our proposal is anchored in multidimensional evidence, including behavioral, neurocognitive, and genetic findings. From a clinical perspective, this work paves the way for dimensional and transdiagnostic approaches, new pharmacological treatments, and educational innovations rooted in a combined neuropsychiatric training. Research-wise, it fosters new models of the social brain and a novel platform to explore the interplay of cognitive and social functions. Finally, we identify new challenges for this synergistic framework.

Attention, in and Out: Scalp-Level and Intracranial EEG Correlates of Interoception and Exteroception.

Interoception, the monitoring of visceral signals, is often presumed to engage attentional mechanisms specifically devoted to inner bodily sensing. In fact, most standardized interoceptive tasks require directing attention to internal signals. However, most studies in the field have failed to compare attentional modulations between internally- and externally-driven processes, thus probing blind to the specificity of the former. Here we address this issue through a multidimensional approach combining behavioral measures, analyses of event-related potentials and functional connectivity via high-density electroencephalography, and intracranial recordings. In Study 1, 50 healthy volunteers performed a heartbeat detection task as we recorded modulations of the heartbeat-evoked potential (HEP) in three conditions: exteroception, basal interoception (also termed interoceptive accuracy), and post-feedback interoception (sometimes called interoceptive learning). In Study 2, to evaluate whether key interoceptive areas (posterior insula, inferior frontal gyrus, amygdala, and somatosensory cortex) were differentially modulated by externally- and internally-driven processes, we analyzed human intracranial recordings with depth electrodes in these regions. This unique technique provides a very fine grained spatio-temporal resolution compared to other techniques, such as EEG or fMRI. We found that both interoceptive conditions in Study 1 yielded greater HEP amplitudes than the exteroceptive one. In addition, connectivity analysis showed that post-feedback interoception, relative to basal interoception, involved enhanced long-distance connections linking frontal and posterior regions. Moreover, results from Study 2 showed a differentiation between oscillations during basal interoception (broadband: 35-110 Hz) and exteroception (1-35 Hz) in the insula, the amygdala, the somatosensory cortex, and the inferior frontal gyrus. In sum, this work provides convergent evidence for the specificity and dynamics of attentional mechanisms involved in interoception.

Enhanced Working Memory Binding by Direct Electrical Stimulation of the Parietal Cortex.

Recent works evince the critical role of visual short-term memory (STM) binding deficits as a clinical and preclinical marker of Alzheimer's disease (AD). These studies suggest a potential role of posterior brain regions in both the neurocognitive deficits of Alzheimer's patients and STM binding in general. Thereupon, we surmised that stimulation of the posterior parietal cortex (PPC) might be a successful approach to tackle working memory deficits in this condition, especially at early stages. To date, no causal evidence exists of the role of the parietal cortex in STM binding. A unique approach to assess this issue is afforded by single-subject direct intracranial electrical stimulation of specific brain regions during a relevant cognitive task. Electrical stimulation has been used both for clinical purposes and to causally probe brain mechanisms. Previous evidence of electrical currents spreading through white matter along well defined functional circuits indicates that visual working memory mechanisms are subserved by a specific widely distributed network. Here, we stimulated the parietal cortex of a subject with intracranial electrodes as he performed the visual STM task. We compared the ensuing results to those from a non-stimulated condition and to the performance of a matched control group. In brief, direct stimulation of the parietal cortex induced a selective improvement in STM. These results, together with previous studies, provide very preliminary but promising ground to examine behavioral changes upon parietal stimulation in AD. We discuss our results regarding: (a) the usefulness of the task to target prodromal stages of AD; (b) the role of a posterior network in STM binding and in AD; and

Convergence of interoception, emotion, and social cognition: A twofold fMRI meta-analysis and lesion approach.

Guided by indirect evidence, recent approaches propose a tripartite crosstalk among interoceptive signaling, emotional regulation, and low-level social cognition. Here we examined the neurocognitive convergence of such domains. First, we performed three meta-analyses of functional magnetic resonance imaging studies to identify which areas are consistently coactivated by these three systems. Multi-level Kernel Density Analysis (MKDA) revealed major overlaps in the right anterior insular and frontotemporal regions (viz., the orbitofrontal and inferior frontal gyri, the amygdala, and mid temporal lobe/subcortical structures). Second, we explored such domains in patients with fronto-insulo-temporal damage. Relative to controls, the patients showed behavioral impairments of interoception, emotional processing, and social cognition, with preservation of other cognitive functions. Convergent results from both studies offer direct support for a model of insular-frontotemporal regions integrating interoception, emotion, and social cognition.

How language flows when movements don't: An automated analysis of spontaneous discourse in Parkinson's disease.

To assess the impact of Parkinson's disease (PD) on spontaneous discourse, we conducted computerized analyses of brief monologues produced by 51 patients and 50 controls. We explored differences in semantic fields (via latent semantic analysis), grammatical choices (using part-of-speech tagging), and word-level repetitions (with graph embedding tools). Although overall output was quantitatively similar between groups, patients relied less heavily on action-related concepts and used more subordinate structures. Also, a classification tool operating on grammatical patterns identified monologues as pertaining to patients or controls with 75% accuracy. Finally, while the incidence of dysfluent word repetitions was similar between groups, it allowed inferring the patients' level of motor impairment with 77% accuracy. Our results highlight the relevance of studying naturalistic discourse features to tap the integrity of neural (and, particularly, motor) networks, beyond the possibilities of standard token-level instruments.

Predictive coding in autism spectrum disorder and attention deficit hyperactivity disorder.

Predictive coding has been proposed as a framework to understand neural processes in neuropsychiatric disorders. We used this approach to describe mechanisms responsible for attentional abnormalities in autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD). We monitored brain dynamics of 59 children (8-15 yr old) who had ASD or ADHD or who were control participants via high-density electroencephalography. We performed analysis at the scalp and source-space levels while participants listened to standard and deviant tone sequences. Through task instructions, we manipulated top-down expectation by presenting expected and unexpected deviant sequences. Children with ASD showed reduced superior frontal cortex (FC) responses to unexpected events but increased dorsolateral prefrontal cortex (PFC) activation to expected events. In contrast, children with ADHD exhibited reduced cortical responses in superior FC to expected events but strong PFC activation to unexpected events. Moreover, neural abnormalities were associated with specific control mechanisms, namely, inhibitory control in ASD and set-shifting in ADHD. Based on the predictive coding account, top-down expectation abnormalities could be attributed to a disproportionate reliance (precision) allocated to prior beliefs in ASD and to sensory input in ADHD.

Heart evoked potential triggers brain responses to natural affective scenes: A preliminary study.

The relationship between ongoing brain interoceptive signals and emotional processes has been addressed only indirectly through external stimulus-locked measures. In this study, an internal body trigger (heart evoked potential, HEP) was used to measure ongoing internally triggered signals during emotional states. We employed high-density electroencephalography (hd-EEG), source reconstruction analysis, and behavioral measures to assess healthy participants watching emotion-inducing video-clips (positive, negative, and neutral emotions). Results showed emotional modulation of the HEP at specific source-space nodes of the fronto-insulo-temporal networks related to affective-cognitive integration. This study is the first to assess the direct convergence among continuous triggers of viscerosensory cortical markers and emotion through dynamic stimuli presentation.

Disentangling interoception: insights from focal strokes affecting the perception of external and internal milieus.

Interoception is the moment-to-moment sensing of the physiological condition of the body. The multimodal sources of interoception can be classified into two different streams of afferents: an internal pathway of signals arising from core structures (i.e., heart, blood vessels, and bronchi) and an external pathway of body-mapped sensations (i.e., chemosensation and pain) arising from peripersonal space. This study examines differential processing along these streams within the insular cortex (IC) and their subcortical tracts connecting frontotemporal networks. Two rare patients presenting focal lesions of the IC (insular lesion, IL) or its subcortical tracts (subcortical lesion, SL) were tested. Internally generated interoceptive streams were assessed through a heartbeat detection (HBD) task, while those externally triggered were tapped via taste, smell, and pain recognition tasks. A differential pattern was observed. The IC patient showed impaired internal signal processing while the SL patient exhibited external perception deficits. Such selective deficits remained even when comparing each patient with a group of healthy controls and a group of brain-damaged patients. These outcomes suggest the existence of distinguishable interoceptive streams. Results are discussed in relation with neuroanatomical substrates, involving a fronto-insulo-temporal network for interoceptive and cognitive contextual integration.

Hypnotic suggestion: a test for the voluntary action problem.

To study voluntary action a dissociation must be established between the somatic event (e.g, motor action) and what the agent voluntarily does (e.g, handing a tool to a friend). We propose that cognitive neuroscience studies of hypnotic suggestion can accomplish this dissociation between action and will (more specifically, between action and intention, or action and volition). Thus, hypnotic suggestion may afford an empirical testing ground to study voluntary action, distinguishing voluntariness from action.