Centering cognitive neuroscience on task demands and generalization.

Cognitive neuroscience seeks generalizable theories explaining the relationship between behavioral, physiological and mental states. In pursuit of such theories, we propose a theoretical and empirical framework that centers on understanding task demands and the mutual constraints they impose on behavior and neural activity. Task demands emerge from the interaction between an agent's sensory impressions, goals and behavior, which jointly shape the activity and structure of the nervous system on multiple spatiotemporal scales. Understanding this interaction requires multitask studies that vary more than one experimental component (for example, stimuli and instructions) combined with dense behavioral and neural sampling and explicit testing for generalization across tasks and data modalities. By centering task demands rather than mental processes that tasks are assumed to engage, this framework paves the way for the discovery of new generalizable concepts unconstrained by existing taxonomies, and moves cognitive neuroscience toward an action-oriented, dynamic and integrated view of the brain.

Hi-Fi fMRI: High-resolution, fast-sampled and sub-second whole-brain functional MRI at 3T in humans.

Functional magnetic resonance imaging (fMRI) is a methodological cornerstone of neuroscience. Most studies measure blood-oxygen-level-dependent (BOLD) signal using echo-planar imaging (EPI), Cartesian sampling, and image reconstruction with a one-to-one correspondence between the number of acquired volumes and reconstructed images. However, EPI schemes are subject to trade-offs between spatial and temporal resolutions. We overcome these limitations by measuring BOLD with a gradient recalled echo (GRE) with 3D radial-spiral phyllotaxis trajectory at a high sampling rate (28.24ms) on standard 3T field-strength. The framework enables the reconstruction of 3D signal time courses with whole-brain coverage at simultaneously higher spatial (1mm 3 ) and temporal (up to 250ms) resolutions, as compared to optimized EPI schemes. Additionally, artifacts are corrected before image reconstruction; the desired temporal resolution is chosen after scanning and without assumptions on the shape of the hemodynamic response. By showing activation in the calcarine sulcus of 20 participants performing an ON-OFF visual paradigm, we demonstrate the reliability of our method for cognitive neuroscience research.

Rapid encoding of task regularities in the human hippocampus guides sensorimotor timing.

The brain encodes the statistical regularities of the environment in a task-specific yet flexible and generalizable format. Here, we seek to understand this process by bridging two parallel lines of research, one centered on sensorimotor timing, and the other on cognitive mapping in the hippocampal system. By combining functional magnetic resonance imaging (fMRI) with a fast-paced time-to-contact (TTC) estimation task, we found that the hippocampus signaled behavioral feedback received in each trial as well as performance improvements across trials along with reward-processing regions. Critically, it signaled performance improvements independent from the tested intervals, and its activity accounted for the trial-wise regression-to-the-mean biases in TTC estimation. This is in line with the idea that the hippocampus supports the rapid encoding of temporal context even on short time scales in a behavior-dependent manner. Our results emphasize the central role of the hippocampus in statistical learning and position it at the core of a brain-wide network updating sensorimotor representations in real time for flexible behavior.

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.

Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence.

Magnetic resonance-based eye tracking using deep neural networks.

Viewing behavior provides a window into many central aspects of human cognition and health, and it is an important variable of interest or confound in many functional magnetic resonance imaging (fMRI) studies. To make eye tracking freely and widely available for MRI research, we developed DeepMReye, a convolutional neural network (CNN) that decodes gaze position from the magnetic resonance signal of the eyeballs. It performs cameraless eye tracking at subimaging temporal resolution in held-out participants with little training data and across a broad range of scanning protocols. Critically, it works even in existing datasets and when the eyes are closed. Decoded eye movements explain network-wide brain activity also in regions not associated with oculomotor function. This work emphasizes the importance of eye tracking for the interpretation of fMRI results and provides an open source software solution that is widely applicable in research and clinical settings.

Interpreting wide-band neural activity using convolutional neural networks.

Rapid progress in technologies such as calcium imaging and electrophysiology has seen a dramatic increase in the size and extent of neural recordings. Even so, interpretation of this data requires considerable knowledge about the nature of the representation and often depends on manual operations. Decoding provides a means to infer the information content of such recordings but typically requires highly processed data and prior knowledge of the encoding scheme. Here, we developed a deep-learning framework able to decode sensory and behavioral variables directly from wide-band neural data. The network requires little user input and generalizes across stimuli, behaviors, brain regions, and recording techniques. Once trained, it can be analyzed to determine elements of the neural code that are informative about a given variable. We validated this approach using electrophysiological and calcium-imaging data from rodent auditory cortex and hippocampus as well as human electrocorticography (ECoG) data. We show successful decoding of finger movement, auditory stimuli, and spatial behaviors - including a novel representation of head direction - from raw neural activity.

Behavior-dependent directional tuning in the human visual-navigation network.

The brain derives cognitive maps from sensory experience that guide memory formation and behavior. Despite extensive efforts, it still remains unclear how the underlying population activity unfolds during spatial navigation and how it relates to memory performance. To examine these processes, we combined 7T-fMRI with a kernel-based encoding model of virtual navigation to map world-centered directional tuning across the human cortex. First, we present an in-depth analysis of directional tuning in visual, retrosplenial, parahippocampal and medial temporal cortices. Second, we show that tuning strength, width and topology of this directional code during memory-guided navigation depend on successful encoding of the environment. Finally, we show that participants' locomotory state influences this tuning in sensory and mnemonic regions such as the hippocampus. We demonstrate a direct link between neural population tuning and human cognition, where high-level memory processing interacts with network-wide visuospatial coding in the service of behavior.

Deforming the metric of cognitive maps distorts memory.

Environmental boundaries anchor cognitive maps that support memory. However, trapezoidal boundary geometry distorts the regular firing patterns of entorhinal grid cells, proposedly providing a metric for cognitive maps. Here we test the impact of trapezoidal boundary geometry on human spatial memory using immersive virtual reality. Consistent with reduced regularity of grid patterns in rodents and a grid-cell model based on the eigenvectors of the successor representation, human positional memory was degraded in a trapezoid environment compared with a square environment-an effect that was particularly pronounced in the narrow part of the trapezoid. Congruent with changes in the spatial frequency of eigenvector grid patterns, distance estimates between remembered positions were persistently biased, revealing distorted memory maps that explained behaviour better than the objective maps. Our findings demonstrate that environmental geometry affects human spatial memory in a similar manner to rodent grid-cell activity and, therefore, strengthen the putative link between grid cells and behaviour along with their cognitive functions beyond navigation.

How the Brain's Navigation System Shapes Our Visual Experience.

We explore the environment not only by navigating, but also by viewing our surroundings with our eyes. Here we review growing evidence that the mammalian hippocampal formation, extensively studied in the context of navigation and memory, mediates a representation of visual space that is stably anchored to the external world. This visual representation puts the hippocampal formation in a central position to guide viewing behavior and to modulate visual processing beyond the medial temporal lobe (MTL). We suggest that vision and navigation share several key computational challenges that are solved by overlapping and potentially common neural systems, making vision an optimal domain to explore whether and how the MTL supports cognitive operations beyond navigation.

Real-motion signals in human early visual cortex.

Eye movements induce visual motion that can complicate the stable perception of the world. The visual system compensates for such self-induced visual motion by integrating visual input with efference copies of eye movement commands. This mechanism is central as it does not only support perceptual stability but also mediates reliable perception of world-centered objective motion. In humans, it remains elusive whether visual motion responses in early retinotopic cortex are driven by objective motion or by retinal motion associated with it. To address this question, we used fMRI to examine functional responses of sixteen visual areas to combinations of planar objective motion and pursuit eye movements. Observers were exposed to objective motion that was faster, matched or slower relative to pursuit, allowing us to compare conditions that differed in objective motion velocity while retinal motion and eye movement signals were matched. Our results show that not only higher level motion regions such as V3A and V6, but also early visual areas signaled the velocity of objective motion, hence the product of integrating retinal with non-retinal signals. These results shed new light on mechanisms that mediate perceptual stability and real-motion perception, and show that extra-retinal signals related to pursuit eye movements influence processing in human early visual cortex.

Hexadirectional coding of visual space in human entorhinal cortex.

Entorhinal grid cells map the local environment, but their involvement beyond spatial navigation remains elusive. We examined human functional MRI responses during a highly controlled visual tracking task and show that entorhinal cortex exhibited a sixfold rotationally symmetric signal encoding gaze direction. Our results provide evidence for a grid-like entorhinal code for visual space and suggest a more general role of the entorhinal grid system in coding information along continuous dimensions.