A hierarchy of linguistic predictions during natural language comprehension.

Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It has been suggested that the brain uses prediction to guide the interpretation of incoming input. However, the role of prediction in language processing remains disputed, with disagreement about both the ubiquity and representational nature of predictions. Here, we address both issues by analyzing brain recordings of participants listening to audiobooks, and using a deep neural network (GPT-2) to precisely quantify contextual predictions. First, we establish that brain responses to words are modulated by ubiquitous predictions. Next, we disentangle model-based predictions into distinct dimensions, revealing dissociable neural signatures of predictions about syntactic category (parts of speech), phonemes, and semantics. Finally, we show that high-level (word) predictions inform low-level (phoneme) predictions, supporting hierarchical predictive processing. Together, these results underscore the ubiquity of prediction in language processing, showing that the brain spontaneously predicts upcoming language at multiple levels of abstraction.

Conceptual Associations Generate Sensory Predictions.

A crucial ability of the human brain is to learn and exploit probabilistic associations between stimuli to facilitate perception and behavior by predicting future events. Although studies have shown how perceptual relationships are used to predict sensory inputs, relational knowledge is often between concepts rather than percepts (e.g., we learned to associate cats with dogs, rather than specific images of cats and dogs). Here, we asked if and how sensory responses to visual input may be modulated by predictions derived from conceptual associations. To this end we exposed participants of both sexes to arbitrary word-word pairs (e.g., car-dog) repeatedly, creating an expectation of the second word, conditional on the occurrence of the first. In a subsequent session, we exposed participants to novel word-picture pairs, while measuring fMRI BOLD responses. All word-picture pairs were equally likely, but half of the pairs conformed to the previously formed conceptual (word-word) associations, whereas the other half violated this association. Results showed suppressed sensory responses throughout the ventral visual stream, including early visual cortex, to pictures that corresponded to the previously expected words compared with unexpected words. This suggests that the learned conceptual associations were used to generate sensory predictions that modulated processing of the picture stimuli. Moreover, these modulations were tuning specific, selectively suppressing neural populations tuned toward the expected input. Combined, our results suggest that recently acquired conceptual priors are generalized across domains and used by the sensory brain to generate category-specific predictions, facilitating processing of expected visual input.SIGNIFICANCE STATEMENT Perceptual predictions play a crucial role in facilitating perception and the integration of sensory information. However, little is known about whether and how the brain uses more abstract, conceptual priors to form sensory predictions. In our preregistered study, we show that priors derived from recently acquired arbitrary conceptual associations result in category-specific predictions that modulate perceptual processing throughout the ventral visual hierarchy, including early visual cortex. These results suggest that the predictive brain uses prior knowledge across various domains to modulate perception, thereby extending our understanding of the extensive role predictions play in perception.

Humans predict the forest, not the trees: statistical learning of spatiotemporal structure in visual scenes.

The human brain is capable of using statistical regularities to predict future inputs. In the real world, such inputs typically comprise a collection of objects (e.g. a forest constitutes numerous trees). The present study aimed to investigate whether perceptual anticipation relies on lower-level or higher-level information. Specifically, we examined whether the human brain anticipates each object in a scene individually or anticipates the scene as a whole. To explore this issue, we first trained participants to associate co-occurring objects within fixed spatial arrangements. Meanwhile, participants implicitly learned temporal regularities between these displays. We then tested how spatial and temporal violations of the structure modulated behavior and neural activity in the visual system using fMRI. We found that participants only showed a behavioral advantage of temporal regularities when the displays conformed to their previously learned spatial structure, demonstrating that humans form configuration-specific temporal expectations instead of predicting individual objects. Similarly, we found suppression of neural responses for temporally expected compared with temporally unexpected objects in lateral occipital cortex only when the objects were embedded within expected configurations. Overall, our findings indicate that humans form expectations about object configurations, demonstrating the prioritization of higher-level over lower-level information in temporal expectation.

Perceptual confirmation bias and decision bias underlie adaptation to sequential regularities.

Our perception does not depend exclusively on the immediate sensory input. It is also influenced by our internal predictions derived from prior observations and the temporal regularities of the environment, which can result in choice history biases. However, it is unclear how this flexible use of prior information to predict the future influences perceptual decisions. Prior information may bias decisions independently of the current sensory input, or it may modulate the weight of current sensory input based on its consistency with the expectation. To address this question, we used a visual decision-making task and manipulated the transitional probabilities between successive noisy grating stimuli. Using a reverse correlation analysis, we evaluated the contribution of stimulus-independent decision bias and stimulus-dependent sensitivity modulations to choice history biases. We found that both effects coexist, whereby there was increased bias to respond in line with the predicted orientation alongside modulations in perceptual sensitivity to favor perceptual information consistent with the prediction, akin to selective attention. Furthermore, at the individual differences level, we investigated the relationship between autistic-like traits and the adaptation of choice history biases to the sequential statistics of the environment. Over two studies, we found no convincing evidence of reduced adaptation to sequential regularities in individuals with high autistic-like traits. In sum, we present robust evidence for both perceptual confirmation bias and decision bias supporting adaptation to sequential regularities in the environment.

Amodal completion across the brain: The impact of structure and knowledge.

This study investigates the phenomenon of amodal completion within the context of naturalistic objects, employing a repetition suppression paradigm to disentangle the influence of structure and knowledge cues on how objects are completed. The research focuses on early visual cortex (EVC) and lateral occipital complex (LOC), shedding light on how these brain regions respond to different completion scenarios. In LOC, we observed suppressed responses to structure and knowledge-compatible stimuli, providing evidence that both cues influence neural processing in higher-level visual areas. However, in EVC, we did not find evidence for differential responses to completions compatible or incompatible with either structural or knowledge-based expectations. Together, our findings suggest that the interplay between structure and knowledge cues in amodal completion predominantly impacts higher-level visual processing, with less pronounced effects on the early visual cortex. This study contributes to our understanding of the complex mechanisms underlying visual perception and highlights the distinct roles played by different brain regions in amodal completion.

Updating Contextual Sensory Expectations for Adaptive Behavior.

The brain has the extraordinary capacity to construct predictive models of the environment by internalizing statistical regularities in the sensory inputs. The resulting sensory expectations shape how we perceive and react to the world; at the neural level, this relates to decreased neural responses to expected than unexpected stimuli ("expectation suppression"). Crucially, expectations may need revision as context changes. However, existing research has often neglected this issue. Further, it is unclear whether contextual revisions apply selectively to expectations relevant to the task at hand, hence serving adaptive behavior. The present fMRI study examined how contextual visual expectations spread throughout the cortical hierarchy as we update our beliefs. We created a volatile environment: two alternating contexts contained different sequences of object images, thereby producing context-dependent expectations that needed revision when the context changed. Human participants of both sexes attended a training session before scanning to learn the contextual sequences. The fMRI experiment then tested for the emergence of contextual expectation suppression in two separate tasks, respectively, with task-relevant and task-irrelevant expectations. Effects of contextual expectation emerged progressively across the cortical hierarchy as participants attuned themselves to the context: expectation suppression appeared first in the insula, inferior frontal gyrus, and posterior parietal cortex, followed by the ventral visual stream, up to early visual cortex. This applied selectively to task-relevant expectations. Together, the present results suggest that an insular and frontoparietal executive control network may guide the flexible deployment of contextual sensory expectations for adaptive behavior in our complex and dynamic world.SIGNIFICANCE STATEMENT The world is structured by statistical regularities, which we use to predict the future. This is often accompanied by suppressed neural responses to expected compared with unexpected events ("expectation suppression"). Crucially, the world is also highly volatile and context-dependent: expected events may become unexpected when the context changes, thus raising the crucial need for belief updating. However, this issue has generally been neglected. By setting up a volatile environment, we show that expectation suppression emerges first in executive control regions, followed by relevant sensory areas, only when observers use their expectations to optimize behavior. This provides surprising yet clear evidence on how the brain controls the updating of sensory expectations for adaptive behavior in our ever-changing world.

Brief Stimuli Cast a Persistent Long-Term Trace in Visual Cortex.

Visual processing is strongly influenced by recent stimulus history, a phenomenon termed adaptation. Prominent theories cast adaptation as a consequence of optimized encoding of visual information by exploiting the temporal statistics of the world. However, this would require the visual system to track the history of individual briefly experienced events, within a stream of visual input, to build up statistical representations over longer timescales. Here, using an openly available dataset from the Allen Brain Observatory, we show that neurons in the early visual cortex of the mouse indeed maintain long-term traces of individual past stimuli that persist despite the presentation of several intervening stimuli, leading to long-term and stimulus-specific adaptation over dozens of seconds. Long-term adaptation was selectively expressed in cortical, but not in thalamic, neurons, which only showed short-term adaptation. Early visual cortex thus maintains concurrent stimulus-specific memory traces of past input, enabling the visual system to build up a statistical representation of the world to optimize the encoding of new information in a changing environment.SIGNIFICANCE STATEMENT In the natural world, previous sensory input is predictive of current input over multisecond timescales. The visual system could exploit these predictabilities by adapting current visual processing to the long-term history of visual input. However, it is unclear whether the visual system can track the history of individual briefly experienced images, within a stream of input, to build up statistical representations over such long timescales. Here, we show that neurons in early visual cortex of the mouse brain exhibit remarkably long-term adaptation to brief stimuli, persisting over dozens of seconds, and despite the presentation of several intervening stimuli. The visual cortex thus maintains long-term traces of individual briefly experienced past images, enabling the formation of statistical representations over extended timescales.

Stubborn Predictions in Primary Visual Cortex.

Perceivers can use past experiences to make sense of ambiguous sensory signals. However, this may be inappropriate when the world changes and past experiences no longer predict what the future holds. Optimal learning models propose that observers decide whether to stick with or update their predictions by tracking the uncertainty or "precision" of their expectations. However, contrasting theories of prediction have argued that we are prone to misestimate uncertainty-leading to stubborn predictions that are difficult to dislodge. To compare these possibilities, we had participants learn novel perceptual predictions before using fMRI to record visual brain activity when predictive contingencies were disrupted-meaning that previously "expected" events became objectively improbable. Multivariate pattern analyses revealed that expected events continued to be decoded with greater fidelity from primary visual cortex, despite marked changes in the statistical structure of the environment, which rendered these expectations no longer valid. These results suggest that our perceptual systems do indeed form stubborn predictions even from short periods of learning-and more generally suggest that top-down expectations have the potential to help or hinder perceptual inference in bounded minds like ours.

Choice Boosts Curiosity.

In our connected era, we spend significant time and effort satisfying our curiosity. Often, we choose which information we seek, but sometimes the selection is made for us. We hypothesized that humans exhibit enhanced curiosity in the context of choice. We designed a task in which healthy participants saw two lotteries on each trial. On some trials, participants chose which lottery to play. On other trials, the lottery was selected for them. Participants then indicated their curiosity about the outcome of the to-be-played lottery via self-report ratings (Experiment 1, N = 34) or willingness-to-wait decisions (Experiment 2, N = 34). We found that participants exhibited higher curiosity ratings and greater willingness to wait for the outcome of lotteries they had chosen than for lotteries that had been selected for them (controlling for initial preference). This demonstrates that choice boosts curiosity, which may have implications for boosting learning, memory, and motivation.

Alpha Oscillations Shape Sensory Representation and Perceptual Sensitivity.

Alpha activity (8-14 Hz) is the dominant rhythm in the awake brain and is thought to play an important role in setting the internal state of the brain. Previous work has associated states of decreased alpha power with enhanced neural excitability. However, evidence is mixed on whether and how such excitability enhancement modulates sensory signals of interest versus noise differently, and what, if any, are the consequences for subsequent perception. Here, human subjects (male and female) performed a visual detection task in which we manipulated their decision criteria in a blockwise manner. Although our manipulation led to substantial criterion shifts, these shifts were not reflected in prestimulus alpha band changes. Rather, lower prestimulus alpha power in occipital-parietal areas improved perceptual sensitivity and enhanced information content decodable from neural activity patterns. Additionally, oscillatory alpha phase immediately before stimulus presentation modulated accuracy. Together, our results suggest that alpha band dynamics modulate sensory signals of interest more strongly than noise.SIGNIFICANCE STATEMENT The internal state of our brain fluctuates, giving rise to variability in perception and action. Neural oscillations, most prominently in the alpha band, have been suggested to play a role in setting this internal state. Here, we show that ongoing alpha band activity in occipital-parietal regions predicts the quality of visual information decodable in neural activity patterns and subsequently the human observer's sensitivity in a visual detection task. Our results provide comprehensive evidence that visual representation is modulated by ongoing alpha band activity and advance our understanding on how, when faced with unchanging external stimuli, internal neural fluctuations influence perception and behavior.

Spatial and Temporal Context Jointly Modulate the Sensory Response within the Ventral Visual Stream.

Both spatial and temporal context play an important role in visual perception and behavior. Humans can extract statistical regularities from both forms of context to help process the present and to construct expectations about the future. Numerous studies have found reduced neural responses to expected stimuli compared with unexpected stimuli, for both spatial and temporal regularities. However, it is largely unclear whether and how these forms of context interact. In the current fMRI study, 33 human volunteers were exposed to pairs of object stimuli that could be expected or surprising in terms of their spatial and temporal context. We found reliable independent contributions of both spatial and temporal context in modulating the neural response. Specifically, neural responses to stimuli in expected compared with unexpected contexts were suppressed throughout the ventral visual stream. These results suggest that both spatial and temporal context may aid sensory processing in a similar fashion, providing evidence on how different types of context jointly modulate perceptual processing.

Temporal prediction elicits rhythmic preactivation of relevant sensory cortices.

Being able to anticipate events before they happen facilitates stimulus processing. The anticipation of the contents of events is thought to be implemented by the elicitation of prestimulus templates in sensory cortex. In contrast, the anticipation of the timing of events is typically associated with entrainment of neural oscillations. It is so far unknown whether and in which conditions temporal expectations interact with feature-based expectations, and, consequently, whether entrainment modulates the generation of content-specific sensory templates. In this study, we investigated the role of temporal expectations in a sensory discrimination task. We presented participants with rhythmically interleaved visual and auditory streams of relevant and irrelevant stimuli while measuring neural activity using magnetoencephalography. We found no evidence that rhythmic stimulation induced prestimulus feature templates. However, we did observe clear anticipatory rhythmic preactivation of the relevant sensory cortices. This oscillatory activity peaked at behaviourally relevant, in-phase, intervals. Our results suggest that temporal expectations about stimulus features do not behave similarly to explicitly cued, nonrhythmic, expectations, yet elicit a distinct form of modality-specific preactivation.

Scene Context Impairs Perception of Semantically Congruent Objects.

Visual scene context is well-known to facilitate the recognition of scene-congruent objects. Interestingly, however, according to predictive-processing accounts of brain function, scene congruency may lead to reduced (rather than enhanced) processing of congruent objects, compared with incongruent ones, because congruent objects elicit reduced prediction-error responses. We tested this counterintuitive hypothesis in two online behavioral experiments with human participants (N = 300). We found clear evidence for impaired perception of congruent objects, both in a change-detection task measuring response times and in a bias-free object-discrimination task measuring accuracy. Congruency costs were related to independent subjective congruency ratings. Finally, we show that the reported effects cannot be explained by low-level stimulus confounds, response biases, or top-down strategy. These results provide convincing evidence for perceptual congruency costs during scene viewing, in line with predictive-processing theory.

Action Enhances Predicted Touch.

It is widely believed that predicted tactile action outcomes are perceptually attenuated. The present experiments determined whether predictive mechanisms necessarily generate attenuation or, instead, can enhance perception-as typically observed in sensory cognition domains outside of action. We manipulated probabilistic expectations in a paradigm often used to demonstrate tactile attenuation. Adult participants produced actions and subsequently rated the intensity of forces on a static finger. Experiment 1 confirmed previous findings that action outcomes are perceived less intensely than passive stimulation but demonstrated more intense perception when active finger stimulation was removed. Experiments 2 and 3 manipulated prediction explicitly and found that expected touch during action is perceived more intensely than unexpected touch. Computational modeling suggested that expectations increase the gain afforded to expected tactile signals. These findings challenge a central tenet of prominent motor control theories and demonstrate that sensorimotor predictions do not exhibit a qualitatively distinct influence on tactile perception.

Adaptation and serial choice bias for low-level visual features are unaltered in autistic adolescents.

Autism spectrum disorder (ASD), or autism, is characterized by social and non-social symptoms, including sensory hyper- and hyposensitivities. A suggestion has been put forward that some of these symptoms could be explained by differences in how sensory information is integrated with its context, including a lower tendency to leverage the past in the processing of new perceptual input. At least two history-dependent effects of opposite directions have been described in the visual perception literature: a repulsive adaptation effect, where perception of a stimulus is biased away from an adaptor stimulus, and an attractive serial choice bias, where perceptual choices are biased toward the previous choice. In this study, we investigated whether autistic participants differed in either bias from typically developing controls (TDs). Sixty-four adolescent participants (31 with ASD, 33 TDs) were asked to categorize oriented line stimuli in two tasks that were designed so that we would induce either adaptation or serial choice bias. Although our tasks successfully induced both biases, in comparing the two groups we found no differences in the magnitude of adaptation nor in the modulation of perceptual choices by the previous choice. In conclusion, we find no evidence of a decreased integration of the past in visual perception of low-level stimulus features in autistic adolescents.

Apparent Motion Induces Activity Suppression in Early Visual Cortex and Impairs Visual Detection.

Apparent motion (AM) is induced when two stationary visual stimuli are presented in alternating sequence. Intriguingly, AM leads to an impaired detectability of stimuli along the AM path (i.e., AM-induced masking). It has been hypothesized that AM triggers an internal representation of a moving object in early visual cortex, which competes with stimulus-evoked representations of visual stimuli on the motion path in early visual cortex of 25 human adults (16 female). We tested this hypothesis by measuring BOLD responses in early visual cortex during the process of AM-induced masking, using fMRI and population receptive field methods. Surprisingly, and counter to our hypothesis, we showed that AM suppressed, rather than increased, BOLD responses along early visual (V1 and V2) representations of the AM path, including regions that were not directly activated by the AM inducer stimuli. This activity suppression of the visual response predicted the subsequent reduction in detectability of the target that appeared in the middle of the AM path. Our data thereby provide direct empirical evidence for suppressive neural mechanisms underlying AM and suggest that illusory motion can render us blind to objects on the motion path by suppressing neural activity at the earliest cortical stages of visual perception.SIGNIFICANCE STATEMENT When two spatially distinct visual objects are presented in alternating sequence, apparent motion (AM) occurs and impairs detectability of stimuli along its path. The underlying mechanism is thought to be that increased activation in human early visual cortex evoked by AM interferes with the representation of the stimulus. Strikingly, however, we show that AM suppresses neural activity along the motion path, and the strength of activity suppression predicts the subsequent behavioral performance decrement in terms of detecting a stimulus along the AM path. Our findings provide empirical evidence for a suppressive, rather than faciliatory, mechanism underlying AM.

Hippocampal and Prefrontal Theta-Band Mechanisms Underpin Implicit Spatial Context Learning.

Humans can rapidly and seemingly implicitly learn to predict typical locations of relevant items when those items are encountered in familiar spatial contexts. Two important questions remain, however, concerning this type of learning: (1) which neural structures and mechanisms are involved in acquiring and exploiting such contextual knowledge? (2) Is this type of learning truly implicit and unconscious? We now answer both these questions after closely examining behavior and recording neural activity using MEG while observers (male and female) were acquiring and exploiting statistical regularities. Computational modeling of behavioral data suggested that, after repeated exposures to a spatial context, participants' behavior was marked by an abrupt switch to an exploitation strategy of the learnt regularities. MEG recordings showed that hippocampus and prefrontal cortex (PFC) were involved in the task and furthermore revealed a striking dissociation: only the initial learning phase was associated with hippocampal theta band activity, while the subsequent exploitation phase showed a shift in theta band activity to the PFC. Intriguingly, the behavioral benefit of repeated exposures to certain scenes was inversely related to explicit awareness of such repeats, demonstrating the implicit nature of the expectations acquired. Together, these findings demonstrate that (1a) hippocampus and PFC play complementary roles in the implicit, unconscious learning and exploitation of spatial statistical regularities; (1b) these mechanisms are implemented in the theta frequency band; and (2) contextual knowledge can indeed be acquired unconsciously, and awareness of such knowledge can even interfere with the exploitation thereof.SIGNIFICANCE STATEMENT Human visual perception is determined not just by the light that strikes our eyes, but also strongly by our prior knowledge and expectations. Such expectations, particularly about where to expect certain objects given scene context, might be learned implicitly and unconsciously, although this is hotly debated. Furthermore, it is unknown which brain mechanisms underpin this type of learning. We now show that, indeed, spatial prior expectations can be learned without awareness; in fact, strikingly, awareness seems to hinder the exploitation of the relevant knowledge. Furthermore, we demonstrate that one brain mechanism (hippocampal theta-band activity) is responsible for learning in these settings, whereas another mechanism (prefrontal theta-band activity) is involved in exploiting the learned associations.

Object Selection by Automatic Spreading of Top-Down Attentional Signals in V1.

What is selected when attention is directed to a specific location of the visual field? Theories of object-based attention have suggested that when spatial attention is directed to part of an object, attention does not simply enhance the attended location but automatically spreads to enhance all locations that comprise the object. Here, we tested this hypothesis by reconstructing the distribution of attention from primary visual cortex (V1) population neuronal activity patterns in 24 human adults (17 female) using functional magnetic resonance imaging (fMRI) and population-based receptive field (prf) mapping. We find that attention spreads from a spatially cued location to the underlying object, and enhances all spatial locations that comprise the object. Importantly, this spreading was also evident when the object was not task relevant. These data suggest that attentional selection automatically operates at an object level, facilitating the reconstruction of coherent objects from fragmented representations in early visual cortex.SIGNIFICANCE STATEMENT Object perception is an astonishing feat of the visual system. When visual information about orientation, shape, and color enters through our eyes, it has yet to be integrated into a coherent representation of an object. But which visual features constitute a single object and which features belong to the background? The brain mechanisms underpinning object perception are yet to be understood. We now demonstrate that one candidate mechanism, the successive activation of all parts of an object, occurs in early visual cortex and results in a detailed representation of the object following Gestalt principles. Furthermore, our results suggest that object selection occurs automatically, without involving voluntary control.

Amodal completion instead of predictive coding can explain activity suppression of early visual cortex during illusory shape perception.

A set of recent neuroimaging studies observed that the perception of an illusory shape can elicit both positive and negative feedback modulations in different parts of the early visual cortex. When three Pac-Men shapes were aligned in such a way that they created an illusory triangle (i.e., the Kanizsa illusion), neural activity in early visual cortex was enhanced in those neurons that had receptive fields that overlapped with the illusory shape but suppressed in neurons whose receptive field overlapped with the Pac-Men inducers. These results were interpreted as congruent with the predictive coding framework, in which neurons in early visual cortex enhance or suppress their activity depending on whether the top-down predictions match the bottom-up sensory inputs. However, there are several plausible alternative explanations for the activity modulations. Here we tested a recent proposal (Moors, 2015) that the activity suppression in early visual cortex during illusory shape perception reflects neural adaptation to perceptually stable input. Namely, the inducers appear perceptually stable during the illusory shape condition (discs on which a triangle is superimposed), but not during the control condition (discs that change into Pac-Men). We examined this hypothesis by manipulating the perceptual stability of inducers. When the inducers could be perceptually interpreted as persistent circles, we replicated the up- and downregulation pattern shown in previous studies. However, when the inducers could not be perceived as persistent circles, we still observed enhanced activity in neurons representing the illusory shape but the suppression of activity in neurons representing the inducers was absent. Thus our results support the hypothesis that the activity suppression in neurons representing the inducers during the Kanizsa illusion is better explained by neural adaptation to perceptually stable input than by reduced prediction error.

Perceptual Expectations Modulate Low-Frequency Activity: A Statistical Learning Magnetoencephalography Study.

Perceptual expectations can change how a visual stimulus is perceived. Recent studies have shown mixed results in terms of whether expectations modulate sensory representations. Here, we used a statistical learning paradigm to study the temporal characteristics of perceptual expectations. We presented participants with pairs of object images organized in a predictive manner and then recorded their brain activity with magnetoencephalography while they viewed expected and unexpected image pairs on the subsequent day. We observed stronger alpha-band (7-14 Hz) activity in response to unexpected compared with expected object images. Specifically, the alpha-band modulation occurred as early as the onset of the stimuli and was most pronounced in left occipito-temporal cortex. Given that the differential response to expected versus unexpected stimuli occurred in sensory regions early in time, our results suggest that expectations modulate perceptual decision-making by changing the sensory response elicited by the stimuli.