Effects of pupil size as manipulated through ipRGC activation on visual processing.

The size of the eyes' pupils determines how much light enters the eye and also how well this light is focused. Through this route, pupil size shapes the earliest stages of visual processing. Yet causal effects of pupil size on vision are poorly understood and rarely studied. Here we introduce a new way to manipulate pupil size, which relies on activation of intrinsically photosensitive retinal ganglion cells (ipRGCs) to induce sustained pupil constriction. We report the effects of both experimentally induced and spontaneous changes in pupil size on visual processing as measured through EEG. We compare these to the effects of stimulus intensity and covert visual attention, because previous studies have shown that these factors all have comparable effects on some common measures of early visual processing, such as detection performance and steady-state visual evoked potentials; yet it is still unclear whether these are superficial similarities, or rather whether they reflect similar underlying processes. Using a mix of neural-network decoding, ERP analyses, and time-frequency analyses, we find that induced pupil size, spontaneous pupil size, stimulus intensity, and covert visual attention all affect EEG responses, mainly over occipital and parietal electrodes, but-crucially-that they do so in qualitatively different ways. Induced and spontaneous pupil-size changes mainly modulate activity patterns (but not overall power or intertrial coherence) in the high-frequency beta range; this may reflect an effect of pupil size on oculomotor activity and/ or visual processing. In addition, spontaneous (but not induced) pupil size tends to correlate positively with intertrial coherence in the alpha band; this may reflect a non-causal relationship, mediated by arousal. Taken together, our findings suggest that pupil size has qualitatively different effects on visual processing from stimulus intensity and covert visual attention. This shows that pupil size as manipulated through ipRGC activation strongly affects visual processing, and provides concrete starting points for further study of this important yet understudied earliest stage of visual processing.

Neural Repetition Suppression Modulates Time Perception: Evidence From Electrophysiology and Pupillometry.

Human time perception is malleable and subject to many biases. For example, it has repeatedly been shown that stimuli that are physically intense or that are unexpected seem to last longer. Two competing hypotheses have been proposed to account for such biases: One states that these temporal illusions are the result of increased levels of arousal that speeds up neural clock dynamics, whereas the alternative "magnitude coding" account states that the magnitude of sensory responses causally modulates perceived durations. Common experimental paradigms used to study temporal biases cannot dissociate between these accounts, as arousal and sensory magnitude covary and modulate each other. Here, we present two temporal discrimination experiments where two flashing stimuli demarcated the start and end of a to-be-timed interval. These stimuli could be either in the same or a different location, which led to different sensory responses because of neural repetition suppression. Crucially, changes and repetitions were fully predictable, which allowed us to explore effects of sensory response magnitude without changes in arousal or surprise. Intervals with changing markers were perceived as lasting longer than those with repeating markers. We measured EEG (Experiment 1) and pupil size (Experiment 2) and found that temporal perception was related to changes in ERPs (P2) and pupil constriction, both of which have been related to responses in the sensory cortex. Conversely, correlates of surprise and arousal (P3 amplitude and pupil dilation) were unaffected by stimulus repetitions and changes. These results demonstrate, for the first time, that sensory magnitude affects time perception even under constant levels of arousal.

Memory for Stimulus Duration Is Not Bound to Spatial Information.

Different theories have been proposed to explain how the human brain derives an accurate sense of time. One specific class of theories, intrinsic clock theories, postulate that temporal information of a stimulus is represented much like other features such as color and location, bound together to form a coherent percept. Here, we explored to what extent this holds for temporal information after it has been perceived and is held in working memory for subsequent comparison. We recorded EEG of participants who were asked to time stimuli at lateral positions of the screen followed by comparison stimuli presented in the center. Using well-established markers of working memory maintenance, we investigated whether the usage of temporal information evoked neural signatures that were indicative of the location where the stimuli had been presented, both during maintenance and during comparison. Behavior and neural measures including the contralateral delay activity, lateralized alpha suppression, and decoding analyses through time all supported the same conclusion: The representation of location was strongly involved during perception of temporal information, but when temporal information was to be used for comparison, it no longer showed a relation to spatial information. These results support a model where the initial perception of a stimulus involves intrinsic computations, but that this information is subsequently translated to a stimulus-independent format to be used to further guide behavior.

Flexible Working Memory Through Selective Gating and Attentional Tagging.

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

The warning stimulus as retrieval cue: The role of associative memory in temporal preparation.

In a warned reaction time task, the warning stimulus (S1) initiates a process of temporal preparation, which promotes a speeded response to the impending target stimulus (S2). According to the multiple trace theory of temporal preparation (MTP), participants learn the timing of S2 by storing a memory trace on each trial, which contains a temporal profile of the events on that trial. On each new trial, S1 serves as a retrieval cue that implicitly and associatively activates memory traces created on earlier trials, which jointly drive temporal preparation for S2. The idea that S1 assumes this role as a retrieval cue was tested across eight experiments, in which two different S1s were associated with two different distributions of S1-S2 intervals: one with predominantly short and one with predominantly long intervals. Experiments differed regarding the S1 features that made up a pair, ranging from highly distinct (e.g., tone and flash) to more similar (e.g., red and green flash) and verbal (i.e., "short" vs "long"). Exclusively for pairs of highly distinct S1s, the results showed that the S1 cue modified temporal preparation, even in participants who showed no awareness of the contingency. This cueing effect persisted in a subsequent transfer phase, in which the contingency between S1 and the timing of S2 was broken - a fact participants were informed of in advance. Together, these findings support the role of S1 as an implicit retrieval cue, consistent with MTP.

It depends on when you look at it: Salience influences eye movements in natural scene viewing and search early in time.

It is generally accepted that salience affects eye movements in simple artificially created search displays. However, no such consensus exists for eye movements in natural scenes, with several reports arguing that it is mostly high-level cognitive factors that control oculomotor behavior in natural scenes. Here, we manipulate the salience distribution across images by decreasing or increasing the contrast in a gradient across the image. We recorded eye movements in an encoding task (Experiment 1) and a visual search task (Experiment 2) and analyzed the relationship between the latency of fixations and subsequent saccade targeting throughout scene viewing. We find that short-latency first saccades are more likely to land on a region of the image with high salience than long-latency and subsequent saccades in both the encoding and visual search tasks. This implies that salience indeed influences oculomotor behavior in natural scenes, albeit on a different timescale than previously reported. We discuss our findings in relation to current theories of saccade control in natural scenes.

A model of curved saccade trajectories: spike rate adaptation in the brainstem as the cause of deviation away.

The trajectory of saccades to a target is often affected whenever there is a distractor in the visual field. Distractors can cause a saccade to deviate towards their location or away from it. The oculomotor mechanisms that produce deviation towards distractors have been thoroughly explored in behavioral, neurophysiological and computational studies. The mechanisms underlying deviation away, on the other hand, remain unclear. Behavioral findings suggest a mechanism of spatially focused, top-down inhibition in a saccade map, and deviation away has become a tool to investigate such inhibition. However, this inhibition hypothesis has little neuroanatomical or neurophysiological support, and recent findings go against it. Here, we propose that deviation away results from an unbalanced saccade drive from the brainstem, caused by spike rate adaptation in brainstem long-lead burst neurons. Adaptation to stimulation in the direction of the distractor results in an unbalanced drive away from it. An existing model of the saccade system was extended with this theory. The resulting model simulates a wide range of findings on saccade trajectories, including findings that have classically been interpreted to support inhibition views. Furthermore, the model replicated the effect of saccade latency on deviation away, but predicted this effect would be absent with large (400 ms) distractor-target onset asynchrony. This prediction was confirmed in an experiment, which demonstrates that the theory both explains classical findings on saccade trajectories and predicts new findings.

Implicitly learning when to be ready: From instances to categories.

There is growing appreciation for the role of long-term memory in guiding temporal preparation in speeded reaction time tasks. In experiments with variable foreperiods between a warning stimulus (S1) and a target stimulus (S2), preparation is affected by foreperiod distributions experienced in the past, long after the distribution has changed. These effects from memory can shape preparation largely implicitly, outside of participants' awareness. Recent studies have demonstrated the associative nature of memory-guided preparation. When distinct S1s predict different foreperiods, they can trigger differential preparation accordingly. Here, we propose that memory-guided preparation allows for another key feature of learning: the ability to generalize across acquired associations and apply them to novel situations. Participants completed a variable foreperiod task where S1 was a unique image of either a face or a scene on each trial. Images of either category were paired with different distributions with predominantly shorter versus predominantly longer foreperiods. Participants displayed differential preparation to never-before seen images of either category, without being aware of the predictive nature of these categories. They continued doing so in a subsequent Transfer phase, after they had been informed that these contingencies no longer held. A novel rolling regression analysis revealed at a fine timescale how category-guided preparation gradually developed throughout the task, and that explicit information about these contingencies only briefly disrupted memory-guided preparation. These results offer new insights into temporal preparation as the product of a largely implicit process governed by associative learning from past experiences.

Addendum: Implicit learning of temporal behavior in complex dynamic environments.

New analyses of the data in this study (Salet et al., 2021, Psychonomic Bulletin & Review, https://doi.org/10.3758/s13423-020-01873-x ) have led us to reinterpret our main finding. Previously, we had attributed better performance for targets appearing at regular intervals versus irregular intervals to "temporal statistical learning." That is, we surmised that this benefit for the regular intervals arises because participants implicitly distilled the regular 3000 ms interval from the otherwise variable environment (i.e., irregular intervals) to predict future (regular) targets. The analyses presented in this Addendum, however, show that this benefit can be attributed to ongoing "temporal preparation" rather than temporal statistical learning.

FMTP: A unifying computational framework of temporal preparation across time scales.

Temporal preparation is the cognitive function that takes place when anticipating future events. This is commonly considered to involve a process that maximizes preparation at time points that yield a high hazard. However, despite their prominence in the literature, hazard-based theories fail to explain the full range of empirical preparation phenomena. Here, we present the formalized multiple trace theory of temporal preparation (fMTP), an integrative model which develops the alternative perspective that temporal preparation results from associative learning. fMTP builds on established computational principles from the domains of interval timing, motor planning, and associative memory. In fMTP, temporal preparation results from associative learning between a representation of time on the one hand and inhibitory and activating motor units on the other hand. Simulations demonstrate that fMTP can explain phenomena across a range of time scales, from sequential effects operating on a time scale of seconds to long-term memory effects occurring over weeks. We contrast fMTP with models that rely on the hazard function and show that fMTP's learning mechanisms are essential to capture the full range of empirical effects. In a critical experiment using a Gaussian distribution of foreperiods, we show the data to be consistent with fMTP's predictions and to deviate from the hazard function. Additionally, we demonstrate how changing fMTP's parameters can account for participant-to-participant variations in preparation. In sum, with fMTP we put forward a unifying computational framework that explains a family of phenomena in temporal preparation that cannot be jointly explained by conventional theoretical frameworks. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Change biases identify the features that drive time perception.

Time perception is malleable, and the perceived duration of stimuli can be strongly affected by the sensory response they evoke. Such "temporal illusions" provide a window on how different sensory systems contribute to our sense of time. Evidence suggests that the sensory response to different features affects time perception to different extents, mediated by the level of arousal or surprise that they evoke. This, however, makes it difficult to disentangle effects of the sensory response itself from the derived arousal or surprise effects. Here, we demonstrate that time perception is differentially affected by different stimulus features when arousal and surprise are kept constant. In four temporal discrimination experiments, participants were presented with empty intervals (1.25 s-2.25 s) marked by two briefly presented visual marker stimuli, and judged whether the duration was longer or shorter than a 1.75 s reference. Markers either repeated or changed along one of six feature dimensions, in a manner fully predictable to participants. Repetitions and changes would modulate sensory response magnitudes due to neural repetition suppression. Results showed that intervals were perceived as longer when markers changed in location, size, or numerosity. Conversely, changes in face identity, orientation or luminance did not affect time perception. These results point to neural and functional selectivity in the way different stimulus features affect time perception. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Implicit learning of temporal behavior in complex dynamic environments.

Humans can automatically detect and learn to exploit repeated aspects (regularities) of the environment. Timing research suggests that such learning is not only used to anticipate what will happen, but also when it will happen. However, in timing experiments, the intervals to be timed are presented in isolation from other stimuli and explicitly cued, contrasting with naturalistic environments in which intervals are embedded in a constant stream of events and individuals are hardly aware of them. It is unclear whether laboratory findings from timing research translate to a more ecologically valid, implicit environment. Here we show in a game-like experiment, specifically designed to measure naturalistic behavior, that participants implicitly use regular intervals to anticipate future events, even when these intervals are constantly interrupted by irregular yet behaviorally relevant events. This finding extends previous research by showing that individuals not only detect such regularities but can also use this knowledge to decide when to act in a complex environment. Furthermore, this finding demonstrates that this type of learning can occur independently from the ordinal sequence of motor actions, which contrasts this work with earlier motor learning studies. Taken together, our results demonstrate that regularities in the time between events are implicitly monitored and used to predict and act on what happens when, thereby showing that laboratory findings from timing research can generalize to naturalistic environments. Additionally, with the development of our game-like experiment, we demonstrate an approach to test cognitive theories in less controlled, ecologically more valid environments.

You prime what you code: The fAIM model of priming of pop-out.

Our visual brain makes use of recent experience to interact with the visual world, and efficiently select relevant information. This is exemplified by speeded search when target- and distractor features repeat across trials versus when they switch, a phenomenon referred to as intertrial priming. Here, we present fAIM, a computational model that demonstrates how priming can be explained by a simple feature-weighting mechanism integrated into an established model of bottom-up vision. In fAIM, such modulations in feature gains are widespread and not just restricted to one or a few features. Consequentially, priming effects result from the overall tuning of visual features to the task at hand. Such tuning allows the model to reproduce priming for different types of stimuli, including for typical stimulus dimensions such as 'color' and for less obvious dimensions such as 'spikiness' of shapes. Moreover, the model explains some puzzling findings from the literature: it shows how priming can be found for target-distractor stimulus relations rather than for their absolute stimulus values per se, without an explicit representation of relations. Similarly, it simulates effects that have been taken to reflect a modulation of priming by an observers' goals-without any representation of goals in the model. We conclude that priming is best considered as a consequence of a general adaptation of the brain to visual input, and not as a peculiarity of visual search.

Hazard versus history: Temporal preparation is driven by past experience.

The hazard function describes the conditional probability that an event will occur at a given moment, given that it has not yet occurred. In warned reaction time tasks, it is a classical finding that the response to a target stimulus is faster as its hazard is higher, which has led to the widespread belief that hazard somehow drives temporal preparation. Alternatively, recent cognitive theories propose that temporal preparation is driven by memory traces of earlier timing experiences. To distinguish between these views, we presented different groups of participants with different distributions of foreperiods between temporal cues and target stimuli. Three experiments revealed clear transfer effects of this manipulation in a test phase where all participants received, after explicit instruction, the same uniform distribution. These findings demonstrate that temporal preparation is driven by past experience, not by current hazard. (PsycINFO Database Record

Timing a week later: The role of long-term memory in temporal preparation.

Temporal preparation has been investigated extensively by manipulating the foreperiod, the interval between a warning stimulus and target stimulus requiring a speeded response. Although such research has revealed many effects of both the duration and distribution of foreperiods on reaction times, the underlying cognitive mechanism is still largely unknown. Here, we test a recent proposal that temporal preparation is driven by the retrieval of memory traces of past experiences from long-term memory rather than by knowledge about upcoming events. Two groups of participants received different foreperiod distributions in an acquisition phase, which was followed a week later by a transfer phase, in which both groups received the same distribution of foreperiods. We found that the effects of the different foreperiod distributions presented in the acquisition phase were still apparent a week later during the transfer phase, as the reaction time patterns of both groups reflected the old distributions. This occurred even though both groups were provided with full information about the change in the distribution of foreperiods at the start of the transfer phase. These findings provide compelling evidence that long-term memory plays an important role in temporal preparation.

Long-term priming of visual search prevails against the passage of time and counteracting instructions.

Studies on intertrial priming have shown that in visual search experiments, the preceding trial automatically affects search performance: facilitating it when the target features repeat and giving rise to switch costs when they change-so-called (short-term) intertrial priming. These effects also occur at longer time scales: When 1 of 2 possible target colors is more frequent during an experiment block, this results in a prolonged and persistent facilitation for the color that was biased, long after the frequency bias is gone-so-called long-term priming. In this study, we explore the robustness of such long-term priming. In Experiment 1, participants were fully informed of the bias and instructed to prioritize the other unbiased color. Despite these instructions, long-term priming of the biased color persisted in this block, suggesting that guidance by long-term priming is an implicit effect. In Experiment 2, long-term priming was built up in 1 experimental session and was then assessed in a second session a week later. Long-term priming persisted across this week, emphasizing that long-term priming is truly a phenomenon of long-term memory. The results support the view that priming results from the automatic and implicit retrieval of memory traces of past trials. (PsycINFO Database Record

Implicit short- and long-term memory direct our gaze in visual search.

Visual attention is strongly affected by the past: both by recent experience and by long-term regularities in the environment that are encoded in and retrieved from memory. In visual search, intertrial repetition of targets causes speeded response times (short-term priming). Similarly, targets that are presented more often than others may facilitate search, even long after it is no longer present (long-term priming). In this study, we investigate whether such short-term priming and long-term priming depend on dissociable mechanisms. By recording eye movements while participants searched for one of two conjunction targets, we explored at what stages of visual search different forms of priming manifest. We found both long- and short- term priming effects. Long-term priming persisted long after the bias was present, and was again found even in participants who were unaware of a color bias. Short- and long-term priming affected the same stage of the task; both biased eye movements towards targets with the primed color, already starting with the first eye movement. Neither form of priming affected the response phase of a trial, but response repetition did. The results strongly suggest that both long- and short-term memory can implicitly modulate feedforward visual processing.

The long and the short of priming in visual search.

Memory affects visual search, as is particularly evident from findings that when target features are repeated from one trial to the next, selection is faster. Two views have emerged on the nature of the memory representations and mechanisms that cause these intertrial priming effects: independent feature weighting versus episodic retrieval of previous trials. Previous research has attempted to disentangle these views focusing on short term effects. Here, we illustrate that the episodic retrieval models make the unique prediction of long-term priming: biasing one target type will result in priming of this target type for a much longer time, well after the bias has disappeared. We demonstrate that such long-term priming is indeed found for the visual feature of color, but only in conjunction search and not in singleton search. Two follow-up experiments showed that it was the kind of search (conjunction versus singleton) and not the difficulty, that determined whether long-term priming occurred. Long term priming persisted unaltered for at least 200 trials, and could not be explained as the result of explicit strategy. We propose that episodic memory may affect search more consistently than previously thought, and that the mechanisms for intertrial priming may be qualitatively different for singleton and conjunction search.

Can a single short-term mechanism account for priming of pop-out?

Trial-to-trial feature repetition speeds response times in pop-out visual search tasks. These priming effects are often ascribed to a short-term memory system. Recently, however, it has been reported that a 'build-up' sequence of repetitions could facilitate responses over 16 trials later - well beyond twice the typically reported time course (Vision Research, 2011, 51, 1972-1978). Here, we first report two replication attempts that yielded little to no support for such long-term priming of pop-out. The results instead fell in line with the predictions of a previously proposed computational model that describes priming as short-lived facilitation that decays over approximately eight trials (Vision Research, 2010, 50, 2110-2115). In the second part of this study, we show that these data are consistent with a simple formulation of decay with a single timescale, and that there is no significant priming beyond eight trials.

Cognitive programs: software for attention's executive.

What are the computational tasks that an executive controller for visual attention must solve? This question is posed in the context of the Selective Tuning model of attention. The range of required computations go beyond top-down bias signals or region-of-interest determinations, and must deal with overt and covert fixations, process timing and synchronization, information routing, memory, matching control to task, spatial localization, priming, and coordination of bottom-up with top-down information. During task execution, results must be monitored to ensure the expected results. This description includes the kinds of elements that are common in the control of any kind of complex machine or system. We seek a mechanistic integration of the above, in other words, algorithms that accomplish control. Such algorithms operate on representations, transforming a representation of one kind into another, which then forms the input to yet another algorithm. Cognitive Programs (CPs) are hypothesized to capture exactly such representational transformations via stepwise sequences of operations. CPs, an updated and modernized offspring of Ullman's Visual Routines, impose an algorithmic structure to the set of attentional functions and play a role in the overall shaping of attentional modulation of the visual system so that it provides its best performance. This requires that we consider the visual system as a dynamic, yet general-purpose processor tuned to the task and input of the moment. This differs dramatically from the almost universal cognitive and computational views, which regard vision as a passively observing module to which simple questions about percepts can be posed, regardless of task. Differing from Visual Routines, CPs explicitly involve the critical elements of Visual Task Executive (vTE), Visual Attention Executive (vAE), and Visual Working Memory (vWM). Cognitive Programs provide the software that directs the actions of the Selective Tuning model of visual attention.