Trying Harder: How Cognitive Effort Sculpts Neural Representations during Working Memory.

While the exertion of mental effort improves performance on cognitive tasks, the neural mechanisms by which motivational factors impact cognition remain unknown. Here, we used fMRI to test how changes in cognitive effort, induced by changes in task difficulty, impact neural representations of working memory (WM). Participants (both sexes) were precued whether WM difficulty would be hard or easy. We hypothesized that hard trials demanded more effort as a later decision required finer mnemonic precision. Behaviorally, pupil size was larger and response times were slower on hard compared with easy trials suggesting our manipulation of effort succeeded. Neurally, we observed robust persistent activity during delay periods in the prefrontal cortex (PFC), especially during hard trials. Yet, details of the memoranda could not be decoded from patterns in prefrontal activity. In the patterns of activity in the visual cortex, however, we found strong decoding of memorized targets, where accuracy was higher on hard trials. To potentially link these across-region effects, we hypothesized that effort, carried by persistent activity in the PFC, impacts the quality of WM representations encoded in the visual cortex. Indeed, we found that the amplitude of delay period activity in the frontal cortex predicted decoded accuracy in the visual cortex on a trial-wise basis. These results indicate that effort-related feedback signals sculpt population activity in the visual cortex, improving mnemonic fidelity.

Wagers for work: Decomposing the costs of cognitive effort.

Some aspects of cognition are more taxing than others. Accordingly, many people will avoid cognitively demanding tasks in favor of simpler alternatives. Which components of these tasks are costly, and how much, remains unknown. Here, we use a novel task design in which subjects request wages for completing cognitive tasks and a computational modeling procedure that decomposes their wages into the costs driving them. Using working memory as a test case, our approach revealed that gating new information into memory and protecting against interference are costly. Critically, other factors, like memory load, appeared less costly. Other key factors which may drive effort costs, such as error avoidance, had minimal influence on wage requests. Our approach is sensitive to individual differences, and could be used in psychiatric populations to understand the true underlying nature of apparent cognitive deficits.

Common Neural Mechanisms Control Attention and Working Memory.

Although previous studies point to qualitative similarities between working memory (WM) and attention, the degree to which these two constructs rely on shared neural mechanisms remains unknown. Focusing on one such potentially shared mechanism, we tested the hypothesis that selecting an item within WM utilizes similar neural mechanisms as selecting a visible item via a shift of attention. We used fMRI and machine learning to decode both the selection among items visually available and the selection among items stored in WM in human subjects (both sexes). Patterns of activity in visual, parietal, and to a lesser extent frontal cortex predicted the locations of the selected items. Critically, these patterns were strikingly interchangeable; classifiers trained on data during attentional selection predicted selection from WM, and classifiers trained on data during selection from memory predicted attentional selection. Using models of voxel receptive fields, we visualized topographic population activity that revealed gain enhancements at the locations of the externally and internally selected items. Our results suggest that selecting among perceived items and selecting among items in WM share a common mechanism. This common mechanism, analogous to a shift of spatial attention, controls the relative gains of neural populations that encode behaviorally relevant information.SIGNIFICANCE STATEMENT How we allocate our attention to external stimuli that we see and to internal representations of stimuli stored in memory might rely on a common mechanism. Supporting this hypothesis, we demonstrated that not only could patterns of human brain activity predict which items were selected during perception and memory, but that these patterns were interchangeable during external and internal selection. Additionally, this generalized selection mechanism operates by changes in the gains of the neural populations both encoding attended sensory representations and storing relevant memory representations.

Stimulation along the anterior-posterior axis of lateral frontal cortex reduces visual serial dependence.

Serial dependence is an attractive pull that recent perceptual history exerts on current judgments. Theory suggests that this bias is due to a form of short-term plasticity prevalent specifically in the frontal lobe. We sought to test the importance of the frontal lobe to serial dependence by disrupting neural activity along its lateral surface during two tasks with distinct perceptual and motor demands. In our first experiment, stimulation of the lateral prefrontal cortex (LPFC) during an oculomotor delayed response task decreased serial dependence only in the first saccade to the target, whereas stimulation posterior to the LPFC decreased serial dependence only in adjustments to eye position after the first saccade. In our second experiment, which used an orientation discrimination task, stimulation anterior to, in, and posterior to the LPFC all caused equivalent decreases in serial dependence. In this experiment, serial dependence occurred only between stimuli at the same location; an alternation bias was observed across hemifields. Frontal stimulation had no effect on the alternation bias. Transcranial magnetic stimulation to parietal cortex had no effect on serial dependence in either experiment. In summary, our experiments provide evidence for both functional differentiation (Experiment 1) and redundancy (Experiment 2) in frontal cortex with respect to serial dependence.

Behavioral Prioritization Enhances Working Memory Precision and Neural Population Gain.

Humans allocate visual working memory (WM) resource according to behavioral relevance, resulting in more precise memories for more important items. Theoretically, items may be maintained by feature-tuned neural populations, where the relative gain of the populations encoding each item determines precision. To test this hypothesis, we compared the amplitudes of delay period activity in the different parts of retinotopic maps representing each of several WM items, predicting the amplitudes would track behavioral priority. Using fMRI, we scanned participants while they remembered the location of multiple items over a WM delay and then reported the location of one probed item using a memory-guided saccade. Importantly, items were not equally probable to be probed (0.6, 0.3, 0.1, 0.0), which was indicated with a precue. We analyzed fMRI activity in 10 visual field maps in occipital, parietal, and frontal cortex known to be important for visual WM. In early visual cortex, but not association cortex, the amplitude of BOLD activation within voxels corresponding to the retinotopic location of visual WM items increased with the priority of the item. Interestingly, these results were contrasted with a common finding that higher-level brain regions had greater delay period activity, demonstrating a dissociation between the absolute amount of activity in a brain area and the activity of different spatially selective populations within it. These results suggest that the distribution of WM resources according to priority sculpts the relative gains of neural populations that encode items, offering a neural mechanism for how prioritization impacts memory precision.

Spatially Specific Working Memory Activity in the Human Superior Colliculus.

Theoretically, working memory (WM) representations are encoded by population activity of neurons with distributed tuning across the stored feature. Here, we leverage computational neuroimaging approaches to map the topographic organization of human superior colliculus (SC) and model how population activity in SC encodes WM representations. We first modeled receptive field properties of voxels in SC, deriving a detailed topographic organization resembling that of the primate SC. Neural activity within human (5 male and 1 female) SC persisted throughout a retention interval of several types of modified memory-guided saccade tasks. Assuming an underlying neural architecture of the SC based on its retinotopic organization, we used an encoding model to show that the pattern of activity in human SC represents locations stored in WM. Our tasks and models allowed us to dissociate the locations of visual targets and the motor metrics of memory-guided saccades from the spatial locations stored in WM, thus confirming that human SC represents true WM information. These data have several important implications. They add the SC to a growing number of cortical and subcortical brain areas that form distributed networks supporting WM functions. Moreover, they specify a clear neural mechanism by which topographically organized SC encodes WM representations.SIGNIFICANCE STATEMENT Using computational neuroimaging approaches, we mapped the topographic organization of human superior colliculus (SC) and modeled how population activity in SC encodes working memory (WM) representations, rather than simpler visual or motor properties that have been traditionally associated with the laminar maps in the primate SC. Together, these data both position the human SC into a distributed network of brain areas supporting WM and elucidate the neural mechanisms by which the SC supports WM.

Population Dynamics of Early Visual Cortex during Working Memory.

Although the content of working memory (WM) can be decoded from the spatial patterns of brain activity in early visual cortex, how populations encode WM representations remains unclear. Here, we address this limitation by using a model-based approach that reconstructs the feature encoded by population activity measured with fMRI. Using this approach, we could successfully reconstruct the locations of memory-guided saccade goals based on the pattern of activity in visual cortex during a memory delay. We could reconstruct the saccade goal even when we dissociated the visual stimulus from the saccade goal using a memory-guided antisaccade procedure. By comparing the spatiotemporal population dynamics, we find that the representations in visual cortex are stable but can also evolve from a representation of a remembered visual stimulus to a prospective goal. Moreover, because the representation of the antisaccade goal cannot be the result of bottom-up visual stimulation, it must be evoked by top-down signals presumably originating from frontal and/or parietal cortex. Indeed, we find that trial-by-trial fluctuations in delay period activity in frontal and parietal cortex correlate with the precision with which our model reconstructed the maintained saccade goal based on the pattern of activity in visual cortex. Therefore, the population dynamics in visual cortex encode WM representations, and these representations can be sculpted by top-down signals from frontal and parietal cortex.

Cortico-cerebellar network involved in saccade adaptation.

Saccade adaptation is the learning process that ensures that vision and saccades remain calibrated. The central nervous system network involved in these adaptive processes remains unclear because of difficulties in isolating the learning process from the correlated visual and motor processes. Here we imaged the human brain during a novel saccade adaptation paradigm that allowed us to isolate neural signals involved in learning independent of the changes in the amplitude of corrective saccades usually correlated with adaptation. We show that the changes in activation in the ipsiversive cerebellar vermis that track adaptation are not driven by the changes in corrective saccades and thus provide critical supporting evidence for previous findings. Similarly, we find that activation in the dorsomedial wall of the contraversive precuneus mirrors the pattern found in the cerebellum. Finally, we identify dorsolateral and dorsomedial cortical areas in the frontal and parietal lobes that encode the retinal errors following inaccurate saccades used to drive recalibration. Together, these data identify a distributed network of cerebellar and cortical areas and their specific roles in oculomotor learning. NEW & NOTEWORTHY The central nervous system constantly learns from errors and adapts to keep visual targets and saccades in registration. We imaged the human brain while the gain of saccades adapted to a visual target that was displaced while the eye was in motion, inducing retinal error. Activity in the cerebellum and precuneus tracked learning, whereas parts of the dorsolateral and dorsomedial frontal and parietal cortex encoded the retinal error used to drive learning.

Multiple component networks support working memory in prefrontal cortex.

Lateral prefrontal cortex (PFC) is regarded as the hub of the brain's working memory (WM) system, but it remains unclear whether WM is supported by a single distributed network or multiple specialized network components in this region. To investigate this problem, we recorded from neurons in PFC while monkeys made delayed eye movements guided by memory or vision. We show that neuronal responses during these tasks map to three anatomically specific modes of persistent activity. The first two modes encode early and late forms of information storage, whereas the third mode encodes response preparation. Neurons that reflect these modes are concentrated at different anatomical locations in PFC and exhibit distinct patterns of coordinated firing rates and spike timing during WM, consistent with distinct networks. These findings support multiple component models of WM and consequently predict distinct failures that could contribute to neurologic dysfunction.

Human Dorsolateral Prefrontal Cortex Is Not Necessary for Spatial Working Memory.

A dominant theory, based on electrophysiological and lesion evidence from nonhuman primate studies, posits that the dorsolateral prefrontal cortex (dlPFC) stores and maintains working memory (WM) representations. Yet, neuroimaging studies have consistently failed to translate these results to humans; these studies normally find that neural activity persists in the human precentral sulcus (PCS) during WM delays. Here, we attempt to resolve this discrepancy. To test the degree to which dlPFC is necessary for WM, we compared the performance of patients with dlPFC lesions and neurologically healthy controls on a memory-guided saccade task that was used in the monkey studies to measure spatial WM. We found that dlPFC damage only impairs the accuracy of memory-guided saccades if the damage impacts the PCS; lesions to dorsolateral dlPFC that spare the PCS have no effect on WM. These results identify the necessary subregion of the frontal cortex for WM and specify how this influential animal model of human cognition must be revised.

Saccade planning evokes topographically specific activity in the dorsal and ventral streams.

Saccade planning may invoke spatially-specific feedback signals that bias early visual activity in favor of top-down goals. We tested this hypothesis by measuring cortical activity at the early stages of the dorsal and ventral visual processing streams. Human subjects maintained saccade plans to (prosaccade) or away (antisaccade) from a spatial location over long memory-delays. Results show that cortical activity persists in early visual cortex at the retinotopic location of upcoming saccade goals. Topographically specific activity persists as early as V1, and activity increases along both dorsal (V3A/B, IPS0) and ventral (hV4, VO1) visual areas. Importantly, activity persists when saccade goals are available only via working memory and when visual targets and saccade goals are spatially disassociated. We conclude that top-down signals elicit retinotopically specific activity in visual cortex both in the dorsal and ventral streams. Such activity may underlie mechanisms that prioritize locations of task-relevant objects.

Human parietal cortex lesions impact the precision of spatial working memory.

The neural mechanisms that support working memory (WM) depend on persistent neural activity. Within topographically organized maps of space in dorsal parietal cortex, spatially selective neural activity persists during WM for location. However, to date, the necessity of these topographic subregions of human parietal cortex for WM remains unknown. To test the causal relationship of these areas to WM, we compared the performance of patients with lesions to topographically organized parietal cortex with those of controls on a memory-guided saccade (MGS) task as well as a visually guided saccade (VGS) task. The MGS task allowed us to measure WM precision continuously with great sensitivity, whereas the VGS task allowed us to control for any deficits in general spatial or visuomotor processing. Compared with controls, patients generated memory-guided saccades that were significantly slower and less accurate, whereas visually guided saccades were unaffected. These results provide key missing evidence for the causal role of topographic areas in human parietal cortex for WM, as well as the neural mechanisms supporting WM.

Visual working memories are abstractions of percepts.

Pioneering studies demonstrating that the contents of visual working memory (WM) can be decoded from the patterns of multivoxel activity in early visual cortex transformed not only how we study WM, but theories of how memories are stored. For instance, the ability to decode the orientation of memorized gratings is hypothesized to depend on the recruitment of the same neural encoding machinery used for perceiving orientations. However, decoding evidence cannot be used to test the so-called sensory recruitment hypothesis without understanding the underlying nature of what is being decoded. Although unknown during WM, during perception decoding the orientation of gratings does not simply depend on activities of orientation tuned neurons. Rather, it depends on complex interactions between the orientation of the grating, the aperture edges, and the topographic structure of the visual map. Here, our goals are to 1) test how these aperture biases described during perception may affect WM decoding, and 2) leverage carefully manipulated visual stimulus properties of gratings to test how sensory-like are WM codes. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases despite having identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Results indicated that fMRI patterns of delay period activity while maintaining the orientation of a grating with one modulator (eg, radial) were interchangeable with patterns while maintaining a grating with the other modulator (eg, angular). We found significant cross-classification in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using a population receptive field model of the visual field maps. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Visual working memories are abstractions of percepts.

During perception, decoding the orientation of gratings depends on complex interactions between the orientation of the grating, aperture edges, and topographic structure of the visual map. Here, we aimed to test how aperture biases described during perception affect working memory (WM) decoding. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases for identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Neural patterns of delay period activity while maintaining the orientation of gratings with one modulator (e.g. radial) were interchangeable with patterns while maintaining gratings with the other modulator (e.g. angular) in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using models of visual field map properties. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Feedback scales the spatial tuning of cortical responses during visual memory.

Perception, working memory, and long-term memory each evoke neural responses in visual cortex, suggesting that memory uses encoding mechanisms shared with perception. While previous research has largely focused on how perception and memory are similar, we hypothesized that responses in visual cortex would differ depending on the origins of the inputs. Using fMRI, we quantified spatial tuning in visual cortex while participants (both sexes) viewed, maintained in working memory, or retrieved from long-term memory a peripheral target. In each of these conditions, BOLD responses were spatially tuned and were aligned with the target's polar angle in all measured visual field maps including V1. As expected given the increasing sizes of receptive fields, polar angle tuning during perception increased in width systematically up the visual hierarchy from V1 to V2, V3, hV4, and beyond. In stark contrast, the widths of tuned responses were broad across the visual hierarchy during working memory and long-term memory, matched to the widths in perception in later visual field maps but much broader in V1. This pattern is consistent with the idea that mnemonic responses in V1 stem from top-down sources. Moreover, these tuned responses when biased (clockwise or counterclockwise of target) predicted matched biases in memory, suggesting that the readout of maintained and reinstated mnemonic responses influences memory guided behavior. We conclude that feedback constrains spatial tuning during memory, where earlier visual maps inherit broader tuning from later maps thereby impacting the precision of memory.

Neural mechanisms of resource allocation in working memory.

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. Higher-priority memoranda were decoded with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization. Remarkably, trialwise variability in the magnitude of delay activity in frontal cortex predicted differences in decoded precision between low and high-priority items in visual cortex. These results suggest a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in visual cortex according to behavioral relevance, thus, identifying a neural mechanism for resource allocation.

Prioritized maps of space in human frontoparietal cortex.

Priority maps are theorized to be composed of large populations of neurons organized topographically into a map of gaze-centered space whose activity spatially tags salient and behaviorally relevant information. Here, we identified four priority map candidates along human posterior intraparietal sulcus (IPS0-IPS3) and two along the precentral sulcus (PCS) that contained reliable retinotopically organized maps of contralateral visual space. Persistent activity increased from posterior-to-anterior IPS areas and from inferior-to-superior PCS areas during the maintenance of a working memory representation, the maintenance of covert attention, and the maintenance of a saccade plan. Moreover, decoders trained to predict the locations on one task (e.g., working memory) cross-predicted the locations on other tasks (e.g., attention) in superior PCS and IPS2, suggesting that these patterns of maintenance activity may be interchangeable across the tasks. Such properties make these two areas in frontal and parietal cortex viable priority map candidates.

Differential roles of the frontal and parietal cortices in the control of saccades.

Although externally as well as internally-guided eye movements allow us to flexibly explore the visual environment, their differential neural mechanisms remain elusive. A better understanding of these neural mechanisms will help us to understand the control of action and to elucidate the nature of cognitive deficits in certain psychiatric populations (e.g., schizophrenia) that show increased latencies in volitional but not visually-guided saccades. Both the superior precentral sulcus (sPCS) and the intraparietal sulcus (IPS) are implicated in the control of eye movements. However, it remains unknown what differential contributions the two areas make to the programming of visually-guided and internally-guided saccades. In this study we tested the hypotheses that sPCS and IPS distinctly encode internally-guided saccades and visually-guided saccades. We scanned subjects with fMRI while they generated visually-guided and internally-guided delayed saccades. We used multi-voxel pattern analysis to test whether patterns of cue related, preparatory and saccade related activation could be used to predict the direction of the planned eye movement. Results indicate that patterns in the human sPCS predicted internally-guided saccades but not visually-guided saccades in all trial periods and patterns in the IPS predicted internally-guided saccades and visually-guided saccades equally well. The results support the hypothesis that the human sPCS and IPS make distinct contributions to the control of volitional eye movements.

Neural population dynamics of human working memory.

The activity of neurons in macaque prefrontal cortex (PFC) persists during working memory (WM) delays, providing a mechanism for memory.1,2,3,4,5,6,7,8,9,10,11 Although theory,11,12 including formal network models,13,14 assumes that WM codes are stable over time, PFC neurons exhibit dynamics inconsistent with these assumptions.15,16,17,18,19 Recently, multivariate reanalyses revealed the coexistence of both stable and dynamic WM codes in macaque PFC.20,21,22,23 Human EEG studies also suggest that WM might contain dynamics.24,25 Nonetheless, how WM dynamics vary across the cortical hierarchy and which factors drive dynamics remain unknown. To elucidate WM dynamics in humans, we decoded WM content from fMRI responses across multiple cortical visual field maps.26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 We found coexisting stable and dynamic neural representations of WM during a memory-guided saccade task. Geometric analyses of neural subspaces revealed that early visual cortex exhibited stronger dynamics than high-level visual and frontoparietal cortex. Leveraging models of population receptive fields, we visualized and made the neural dynamics interpretable. We found that during WM delays, V1 population initially encoded a narrowly tuned bump of activation centered on the peripheral memory target. Remarkably, this bump then spread inward toward foveal locations, forming a vector along the trajectory of the forthcoming memory-guided saccade. In other words, the neural code transformed into an abstraction of the stimulus more proximal to memory-guided behavior. Therefore, theories of WM must consider both sensory features and their task-relevant abstractions because changes in the format of memoranda naturally drive neural dynamics.

Mnemonic representations in human lateral geniculate nucleus.

There is a growing appreciation for the role of the thalamus in high-level cognition. Motivated by findings that internal cognitive state drives activity in feedback layers of primary visual cortex (V1) that target the lateral geniculate nucleus (LGN), we investigated the role of LGN in working memory (WM). Specifically, we leveraged model-based neuroimaging approaches to test the hypothesis that human LGN encodes information about spatial locations temporarily encoded in WM. First, we localized and derived a detailed topographic organization in LGN that accords well with previous findings in humans and non-human primates. Next, we used models constructed on the spatial preferences of LGN populations in order to reconstruct spatial locations stored in WM as subjects performed modified memory-guided saccade tasks. We found that population LGN activity faithfully encoded the spatial locations held in memory in all subjects. Importantly, our tasks and models allowed us to dissociate the locations of retinal stimulation and the motor metrics of memory-guided saccades from the maintained spatial locations, thus confirming that human LGN represents true WM information. These findings add LGN to the growing list of subcortical regions involved in WM, and suggest a key pathway by which memories may influence incoming processing at the earliest levels of the visual hierarchy.