A Neural Decision Signal during Internal Sampling from Working Memory in Humans.

How humans transform sensory information into decisions that steer purposeful behavior is a central question in psychology and neuroscience that is traditionally investigated during the sampling of external environmental signals. The decision-making framework of gradual information sampling toward a decision has also been proposed to apply when sampling internal sensory evidence from working memory. However, neural evidence for this proposal remains scarce. Here we show (using scalp EEG in male and female human volunteers) that sampling internal visual representations from working memory elicits a scalp EEG potential associated with gradual evidence accumulation-the central parietal positivity. Consistent with an evolving decision process, we show how this signal (1) scales with the time participants require to reach a decision about the cued memory content and (2) is amplified when having to decide among multiple contents in working memory. These results bring the electrophysiology of decision-making into the domain of working memory and suggest that variability in memory-guided behavior may be driven (at least in part) by variations in the sampling of our inner mental contents.

Time for Memories.

The ability to store information about the past to dynamically predict and prepare for the future is among the most fundamental tasks the brain performs. To date, the problems of understanding how the brain stores and organizes information about the past (memory) and how the brain represents and processes temporal information for adaptive behavior have generally been studied as distinct cognitive functions. This Symposium explores the inherent link between memory and temporal cognition, as well as the potential shared neural mechanisms between them. We suggest that working memory and implicit timing are interconnected and may share overlapping neural mechanisms. Additionally, we explore how temporal structure is encoded in associative and episodic memory and, conversely, the influences of episodic memory on subsequent temporal anticipation and the perception of time. We suggest that neural sequences provide a general computational motif that contributes to timing and working memory, as well as the spatiotemporal coding and recall of episodes.

Multiple and Dissociable Effects of Sensory History on Working-Memory Performance.

Behavioral reports of sensory information are biased by stimulus history. The nature and direction of such serial-dependence biases can differ between experimental settings; both attractive and repulsive biases toward previous stimuli have been observed. How and when these biases arise in the human brain remains largely unexplored. They could occur either via a change in sensory processing itself and/or during postperceptual processes such as maintenance or decision-making. To address this, we tested 20 participants (11 female) and analyzed behavioral and magnetoencephalographic (MEG) data from a working-memory task in which participants were sequentially presented with two randomly oriented gratings, one of which was cued for recall at the end of the trial. Behavioral responses showed evidence for two distinct biases: (1) a within-trial repulsive bias away from the previously encoded orientation on the same trial, and (2) a between-trial attractive bias toward the task-relevant orientation on the previous trial. Multivariate classification of stimulus orientation revealed that neural representations during stimulus encoding were biased away from the previous grating orientation, regardless of whether we considered the within-trial or between-trial prior orientation, despite opposite effects on behavior. These results suggest that repulsive biases occur at the level of sensory processing and can be overridden at postperceptual stages to result in attractive biases in behavior.SIGNIFICANCE STATEMENT Recent experience biases behavioral reports of sensory information, possibly capitalizing on the temporal regularity in our environment. It is still unclear at what stage of stimulus processing such serial biases arise. Here, we recorded behavior and neurophysiological [magnetoencephalographic (MEG)] data to test whether neural activity patterns during early sensory processing show the same biases seen in participants' reports. In a working-memory task that produced multiple biases in behavior, responses were biased toward previous targets, but away from more recent stimuli. Neural activity patterns were uniformly biased away from all previously relevant items. Our results contradict proposals that all serial biases arise at an early sensory processing stage. Instead, neural activity exhibited mostly adaptation-like responses to recent stimuli.

Evidence Accumulation Rate Moderates the Relationship between Enriched Environment Exposure and Age-Related Response Speed Declines.

Older adults exposed to enriched environments (EEs) maintain relatively higher levels of cognitive function, even in the face of compromised markers of brain health. Response speed (RS) is often used as a simple proxy to measure the preservation of global cognitive function in older adults. However, it is unknown which specific selection, decision, and/or motor processes provide the most specific indices of neurocognitive health. Here, using a simple decision task with electroencephalography (EEG), we found that the efficiency with which an individual accumulates sensory evidence was a critical determinant of the extent to which RS was preserved in older adults (63% female, 37% male). Moreover, the mitigating influence of EE on age-related RS declines was most pronounced when evidence accumulation rates were shallowest. These results suggest that the phenomenon of cognitive reserve, whereby high EE individuals can better tolerate suboptimal brain health to facilitate the preservation of cognitive function, is not just applicable to neuroanatomical indicators of brain aging but can be observed in markers of neurophysiology. Our results suggest that EEG metrics of evidence accumulation may index neurocognitive vulnerability of the aging brain.Significance Statement Response speed in older adults is closely linked with trajectories of cognitive aging. Here, by recording brain activity while individuals perform a simple computer task, we identify a neural metric that is a critical determinant of response speed. Older adults exposed to greater cognitive and social stimulation throughout a lifetime could maintain faster responding, even when this neural metric was impaired. This work suggests EEG is a useful technique for interrogating how a lifetime of stimulation benefits brain health in aging.

Turning Attention Inside Out: How Working Memory Serves Behavior.

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Heading Direction Tracks Internally Directed Selective Attention in Visual Working Memory.

We shift our gaze even when we orient attention internally to visual representations in working memory. Here, we show the bodily orienting response associated with internal selective attention is widespread as it also includes the head. In three virtual reality experiments, participants remembered 2 visual items. After a working memory delay, a central color cue indicated which item needed to be reproduced from memory. After the cue, head movements became biased in the direction of the memorized location of the cued memory item-despite there being no items to orient toward in the external environment. The heading-direction bias had a distinct temporal profile from the gaze bias. Our findings reveal that directing attention within the spatial layout of visual working memory bears a strong relation to the overt head orienting response we engage when directing attention to sensory information in the external environment. The heading-direction bias further demonstrates common neural circuitry is engaged during external and internal orienting of attention.

Anticipated moments: temporal structure in attention.

We have come to recognize the brain as a predictive organ, anticipating attributes of the incoming sensory stimulation to guide perception and action in the service of adaptive behaviour. In the quest to understand the neural bases of the modulatory prospective signals that prioritize and select relevant events during perception, one fundamental dimension has until recently been largely overlooked: time. In this Review, we introduce the burgeoning field of temporal attention and illustrate how the brain makes use of various forms of temporal regularities in the environment to guide adaptive behaviour and influence neural processing.

Goal-directed and stimulus-driven selection of internal representations.

Adaptive behavior relies on the selection of relevant sensory information from both the external environment and internal memory representations. In understanding external selection, a classic distinction is made between voluntary (goal-directed) and involuntary (stimulus-driven) guidance of attention. We have developed a task-the anti-retrocue task-to separate and examine voluntary and involuntary guidance of attention to internal representations in visual working memory. We show that both voluntary and involuntary factors influence memory performance but do so in distinct ways. Moreover, by tracking gaze biases linked to attentional focusing in memory, we provide direct evidence for an involuntary "retro-capture" effect whereby external stimuli involuntarily trigger the selection of feature-matching internal representations. We show that stimulus-driven and goal-directed influences compete for selection in memory, and that the balance of this competition-as reflected in oculomotor signatures of internal attention-predicts the quality of ensuing memory-guided behavior. Thus, goal-directed and stimulus-driven factors together determine the fate not only of perception, but also of internal representations in working memory.

Rhythmic Modulation of Visual Perception by Continuous Rhythmic Auditory Stimulation.

At any given moment our sensory systems receive multiple, often rhythmic, inputs from the environment. Processing of temporally structured events in one sensory modality can guide both behavioral and neural processing of events in other sensory modalities, but whether this occurs remains unclear. Here, we used human electroencephalography (EEG) to test the cross-modal influences of a continuous auditory frequency-modulated (FM) sound on visual perception and visual cortical activity. We report systematic fluctuations in perceptual discrimination of brief visual stimuli in line with the phase of the FM-sound. We further show that this rhythmic modulation in visual perception is related to an accompanying rhythmic modulation of neural activity recorded over visual areas. Importantly, in our task, perceptual and neural visual modulations occurred without any abrupt and salient onsets in the energy of the auditory stimulation and without any rhythmic structure in the visual stimulus. As such, the results provide a critical validation for the existence and functional role of cross-modal entrainment and demonstrates its utility for organizing the perception of multisensory stimulation in the natural environment.SIGNIFICANCE STATEMENT Our sensory environment is filled with rhythmic structures that are often multi-sensory in nature. Here, we show that the alignment of neural activity to the phase of an auditory frequency-modulated (FM) sound has cross-modal consequences for vision: yielding systematic fluctuations in perceptual discrimination of brief visual stimuli that are mediated by accompanying rhythmic modulation of neural activity recorded over visual areas. These cross-modal effects on visual neural activity and perception occurred without any abrupt and salient onsets in the energy of the auditory stimulation and without any rhythmic structure in the visual stimulus. The current work shows that continuous auditory fluctuations in the natural environment can provide a pacing signal for neural activity and perception across the senses.

Opening Questions in Visual Working Memory.

In this reflective piece on visual working memory, I depart from the laboriously honed skills of writing a review. Instead of integrating approaches, synthesizing evidence, and building a cohesive perspective, I scratch my head and share niggles and puzzlements. I expose where my scholarship and understanding are stumped by findings and standard views in the literature.

Be there on time: Spatial-temporal regularities guide young children's attention in dynamic environments.

Children's ability to benefit from spatiotemporal regularities to detect goal-relevant targets was tested in a dynamic, extended context. Young adults and children (from a low-deprivation area school in the United Kingdom; N = 80; 5-6 years; 39 female; ethics approval did not permit individual-level race/ethnicity surveying) completed a dynamic visual-search task. Targets and distractors faded in and out of a display over seconds. Half of the targets appeared at predictable times and locations. Search performance in children was poorer overall. Nevertheless, they benefitted equivalently from spatiotemporal regularities, detecting more predictable than unpredictable targets. Children's benefits from predictions correlated positively with their attention. The study brings ecological validity to the study of attentional guidance in children, revealing striking behavioral benefits of dynamic experience-based predictions.

Modulation of the pupillary response by the content of visual working memory.

Studies of selective attention during perception have revealed modulation of the pupillary response according to the brightness of task-relevant (attended) vs. -irrelevant (unattended) stimuli within a visual display. As a strong test of top-down modulation of the pupil response by selective attention, we asked whether changes in pupil diameter follow internal shifts of attention to memoranda of visual stimuli of different brightness maintained in working memory, in the absence of any visual stimulation. Across 3 studies, we reveal dilation of the pupil when participants orient attention to the memorandum of a dark grating relative to that of a bright grating. The effect occurs even when the attention-orienting cue is independent of stimulus brightness, and even when stimulus brightness is merely incidental and not required for the working-memory task of judging stimulus orientation. Furthermore, relative dilation and constriction of the pupil occurred dynamically and followed the changing temporal expectation that 1 or the other stimulus would be probed across the retention delay. The results provide surprising and consistent evidence that pupil responses are under top-down control by cognitive factors, even when there is no direct adaptive gain for such modulation, since no visual stimuli were presented or anticipated. The results also strengthen the view of sensory recruitment during working memory, suggesting even activation of sensory receptors. The thought-provoking corollary to our findings is that the pupils provide a reliable measure of what is in the focus of mind, thus giving a different meaning to old proverbs about the eyes being a window to the mind.

Purpose-Dependent Consequences of Temporal Expectations Serving Perception and Action.

Temporal expectations enable anticipatory brain states that prepare us for upcoming perception and action. We investigated the purpose-dependent nature and consequences of cued temporal expectations on brain and behavior in male and female human volunteers, using two matched visual-motor tasks that stressed either response speed or visual accuracy. We show that the consequences of temporal expectations are fundamentally purpose dependent. Temporal expectations predominantly affected response times when visual demands were low and speed was more important, but perceptual accuracy when visual demands were more challenging. Using magnetoencephalography, we further show how temporal expectations latch onto anticipatory neural states associated with concurrent spatial expectations-modulating task-specific anticipatory neural lateralization of oscillatory brain activity in a modality- and frequency-specific manner. By relating these brain states to behavior, we finally reveal how the behavioral relevance of such anticipatory brain states is similarly purpose dependent.SIGNIFICANCE STATEMENT Knowing when events may occur helps to prepare neural activity for upcoming perception and action. It is becoming increasingly clear that distinct sources of temporal expectations may facilitate performance via distinct mechanisms. Another relevant dimension to consider regards the distinct purposes that temporal expectations may serve. Here, we demonstrate that the consequences of temporal expectations on neurophysiological brain activity and behavior are fundamentally purpose dependent, and show how temporal expectations interact with task-relevant neural states in a modality- and frequency-specific manner. This brings the important insight that the ways in which temporal expectations influence brain and behavior, and how brain activity is related to behavior, are not fixed properties but rather depend on the task at hand.

One Thing Leads to Another: Anticipating Visual Object Identity Based on Associative-Memory Templates.

Probabilistic associations between stimuli afford memory templates that guide perception through proactive anticipatory mechanisms. A great deal of work has examined the behavioral consequences and human electrophysiological substrates of anticipation following probabilistic memory cues that carry spatial or temporal information to guide perception. However, less is understood about the electrophysiological substrates linked to anticipating the sensory content of events based on recurring associations between successive events. Here, we demonstrate behavioral and electrophysiological signatures of using associative-memory templates to guide perception, while equating spatial and temporal anticipation (experiments 1 and 2), as well as target probability and response demands (experiment 2). By recording the electroencephalogram in the two experiments (N = 55; 24 females), we show that two markers in human electrophysiology implicated in spatial and temporal anticipation also contribute to the anticipation of perceptual identity, as follows: attenuation of alpha-band oscillations and the contingent negative variation (CNV). Together, our results show that memory-guided identity templates proactively impact perception and are associated with anticipatory states of attenuated alpha oscillations and the CNV. Furthermore, by isolating object-identity anticipation from spatial and temporal anticipation, our results suggest a role for alpha attenuation and the CNV in specific visual content anticipation beyond general changes in neural excitability or readiness.SIGNIFICANCE STATEMENT Probabilistic associations between stimuli afford memory templates that guide perception through proactive anticipatory mechanisms. The current work isolates the behavioral benefits and electrophysiological signatures of memory-guided identity-based anticipation, while equating anticipation of space, time, motor responses, and task relevance. Our results show that anticipation of the specific identity of a forthcoming percept impacts performance and is associated with states of attenuated alpha oscillations and the contingent negative variation, extending previous work implicating these neural substrates in spatial and temporal preparatory attention. Together, this work bridges fields of attention, memory, and perception, providing new insights into the neural mechanisms that support complex attentional templates.

Decoding visual colour from scalp electroencephalography measurements.

Recent advances have made it possible to decode various aspects of visually presented stimuli from patterns of scalp EEG measurements. As of recently, such multivariate methods have been commonly used to decode visual-spatial features such as location, orientation, or spatial frequency. In the current study, we show that it is also possible to track visual colour processing by using Linear Discriminant Analysis on patterns of EEG activity. Building on other recent demonstrations, we show that colour decoding: (1) reflects sensory qualities (as opposed to, for example, verbal labelling) with a prominent contribution from posterior electrodes contralateral to the stimulus, (2) conforms to a parametric coding space, (3) is possible in multi-item displays, and (4) is comparable in magnitude to the decoding of visual stimulus orientation. Through subsampling our data, we also provide an estimate of the approximate number of trials and participants required for robust decoding. Finally, we show that while colour decoding can be sensitive to subtle differences in luminance, our colour decoding results are primarily driven by measured colour differences between stimuli. Colour decoding opens a relevant new dimension in which to track visual processing using scalp EEG measurements, while bypassing potential confounds associated with decoding approaches that focus on spatial features.

Within-cycle instantaneous frequency profiles report oscillatory waveform dynamics.

The nonsinusoidal waveform is emerging as an important feature of neuronal oscillations. However, the role of single-cycle shape dynamics in rapidly unfolding brain activity remains unclear. Here, we develop an analytical framework that isolates oscillatory signals from time series using masked empirical mode decomposition to quantify dynamical changes in the shape of individual cycles (along with amplitude, frequency, and phase) with instantaneous frequency. We show how phase-alignment, a process of projecting cycles into a regularly sampled phase grid space, makes it possible to compare cycles of different durations and shapes. "Normalized shapes" can then be constructed with high temporal detail while accounting for differences in both duration and amplitude. We find that the instantaneous frequency tracks nonsinusoidal shapes in both simulated and real data. Notably, in local field potential recordings of mouse hippocampal CA1, we find that theta oscillations have a stereotyped slow-descending slope in the cycle-wise average yet exhibit high variability on a cycle-by-cycle basis. We show how principal component analysis allows identification of motifs of theta cycle waveform that have distinct associations to cycle amplitude, cycle duration, and animal movement speed. By allowing investigation into oscillation shape at high temporal resolution, this analytical framework will open new lines of inquiry into how neuronal oscillations support moment-by-moment information processing and integration in brain networks.NEW & NOTEWORTHY We propose a novel analysis approach quantifying nonsinusoidal waveform shape. The approach isolates oscillations with empirical mode decomposition before waveform shape is quantified using phase-aligned instantaneous frequency. This characterizes the full shape profile of individual cycles while accounting for between-cycle differences in duration, amplitude, and timing. We validated in simulations before applying to identify a range of data-driven nonsinusoidal shape motifs in hippocampal theta oscillations.

Rhythmic Temporal Expectation Boosts Neural Activity by Increasing Neural Gain.

Temporal orienting improves sensory processing, akin to other top-down biases. However, it is unknown whether these improvements reflect increased neural gain to any stimuli presented at expected time points, or specific tuning to task-relevant stimulus aspects. Furthermore, while other top-down biases are selective, the extent of trade-offs across time is less well characterized. Here, we tested whether gain and/or tuning of auditory frequency processing in humans is modulated by rhythmic temporal expectations, and whether these modulations are specific to time points relevant for task performance. Healthy participants (N = 23) of either sex performed an auditory discrimination task while their brain activity was measured using magnetoencephalography/electroencephalography (M/EEG). Acoustic stimulation consisted of sequences of brief distractors interspersed with targets, presented in a rhythmic or jittered way. Target rhythmicity not only improved behavioral discrimination accuracy and M/EEG-based decoding of targets, but also of irrelevant distractors preceding these targets. To explain this finding in terms of increased sensitivity and/or sharpened tuning to auditory frequency, we estimated tuning curves based on M/EEG decoding results, with separate parameters describing gain and sharpness. The effect of rhythmic expectation on distractor decoding was linked to gain increase only, suggesting increased neural sensitivity to any stimuli presented at relevant time points.SIGNIFICANCE STATEMENT Being able to predict when an event may happen can improve perception and action related to this event, likely due to the alignment of neural activity to the temporal structure of stimulus streams. However, it is unclear whether rhythmic increases in neural sensitivity are specific to task-relevant targets, and whether they competitively impair stimulus processing at unexpected time points. By combining magnetoencephalography and encephalographic recordings, neural decoding of auditory stimulus features, and modeling, we found that rhythmic expectation improved neural decoding of both relevant targets and irrelevant distractors presented and expected time points, but did not competitively impair stimulus processing at unexpected time points. Using a quantitative model, these results were linked to nonspecific neural gain increases due to rhythmic expectation.

Temporal Expectations Prepare Visual Working Memory for Behavior.

Working memory enables us to retain past sensations in service of anticipated task demands. How we prepare for anticipated task demands during working memory retention remains poorly understood. Here, we focused on the role of time-asking how temporal expectations help prepare for ensuing memory-guided behavior. We manipulated the expected probe time in a delayed change-detection task and report that temporal expectation can have a profound influence on memory-guided behavioral performance. EEG measurements corroborated the utilization of temporal expectations: demonstrating the involvement of a classic EEG signature of temporal expectation-the contingent negative variation-in the context of working memory. We also report the influence of temporal expectations on 2 EEG signatures associated with visual working memory-the lateralization of 8- to 12-Hz alpha activity, and the contralateral delay activity. We observed a dissociation between these signatures, whereby alpha lateralization (but not the contralateral delay activity) adapted to the time of expected memory utilization. These data show how temporal expectations prepare visual working memory for behavior and shed new light on the electrophysiological markers of both temporal expectation and working memory.

Task relevance modulates the behavioural and neural effects of sensory predictions.

The brain is thought to generate internal predictions to optimize behaviour. However, it is unclear whether predictions signalling is an automatic brain function or depends on task demands. Here, we manipulated the spatial/temporal predictability of visual targets, and the relevance of spatial/temporal information provided by auditory cues. We used magnetoencephalography (MEG) to measure participants' brain activity during task performance. Task relevance modulated the influence of predictions on behaviour: spatial/temporal predictability improved spatial/temporal discrimination accuracy, but not vice versa. To explain these effects, we used behavioural responses to estimate subjective predictions under an ideal-observer model. Model-based time-series of predictions and prediction errors (PEs) were associated with dissociable neural responses: predictions correlated with cue-induced beta-band activity in auditory regions and alpha-band activity in visual regions, while stimulus-bound PEs correlated with gamma-band activity in posterior regions. Crucially, task relevance modulated these spectral correlates, suggesting that current goals influence PE and prediction signalling.

Temporally Unconstrained Decoding Reveals Consistent but Time-Varying Stages of Stimulus Processing.

In this article, we propose a method to track trial-specific neural dynamics of stimulus processing and decision making with high temporal precision. By applying this novel method to a perceptual template-matching task, we tracked representational brain states associated with the cascade of neural processing, from early sensory areas to higher order areas that are involved in integration and decision making. We address a major limitation of the traditional decoding approach: that it relies on consistent timing of these processes over trials. Using a TUDA approach, we found that the timing of the cognitive processes involved in perceptual judgments can vary considerably over trials. This revealed that the sequence of processing states was consistent for all subjects and trials, even when the timing of these states varied. Furthermore, we found that the specific timing of states on each trial was related to the quality of performance over trials. Altogether, this work not only highlights the serious pitfalls and misleading interpretations that result from assuming stimulus processing to be synchronous across trials but can also open important avenues to investigate learning and quantify plasticity.