Do actions structure auditory memory? Action-based event segmentation effects on sensory responses, pupil dilation and sequential memory.

Our actions shape our everyday experience: what we experience, how we perceive, and remember it are deeply affected by how we interact with the world. Performing an action to deliver a stimulus engages neurophysiological processes which are reflected in the modulation of sensory and pupil responses. We hypothesized that these processes shape memory encoding, parsing the experience by grouping self- and externally generated stimuli into differentiated events. Participants encoded sound sequences, in which either the first or last few sounds were self-generated and the rest externally generated. We tested recall of the sequential order of sounds that had originated from the same (within event) or different sources (across events). Memory performance was not higher for within-event sounds, suggesting that actions did not structure the memory representation. However, during encoding, we observed the expected electrophysiological response attenuation for self-generated sounds, together with increased pupil dilation triggered by actions. Moreover, at the boundary between events, physiological responses to the first sound from the new source were influenced by the direction of the source switch. Our results suggest that introducing actions creates a stronger contextual shift than removing them, even though actions do not directly contribute to memory performance. This study contributes to our understanding of how interacting with sensory input shapes experiences by exploring the relationships between action effects on sensory responses, pupil dilation, and memory encoding. Importantly, it challenges the notion of a meaningful contribution from low-level neurophysiological mechanisms associated with action execution in the modulation of the self-generation effect.

Attention and prediction in human audition: a lesson from cognitive psychophysiology.

Attention is a hypothetical mechanism in the service of perception that facilitates the processing of relevant information and inhibits the processing of irrelevant information. Prediction is a hypothetical mechanism in the service of perception that considers prior information when interpreting the sensorial input. Although both (attention and prediction) aid perception, they are rarely considered together. Auditory attention typically yields enhanced brain activity, whereas auditory prediction often results in attenuated brain responses. However, when strongly predicted sounds are omitted, brain responses to silence resemble those elicited by sounds. Studies jointly investigating attention and prediction revealed that these different mechanisms may interact, e.g. attention may magnify the processing differences between predicted and unpredicted sounds. Following the predictive coding theory, we suggest that prediction relates to predictions sent down from predictive models housed in higher levels of the processing hierarchy to lower levels and attention refers to gain modulation of the prediction error signal sent up to the higher level. As predictions encode contents and confidence in the sensory data, and as gain can be modulated by the intention of the listener and by the predictability of the input, various possibilities for interactions between attention and prediction can be unfolded. From this perspective, the traditional distinction between bottom-up/exogenous and top-down/endogenous driven attention can be revisited and the classic concepts of attentional gain and attentional trace can be integrated.

Hearing silences: human auditory processing relies on preactivation of sound-specific brain activity patterns.

The remarkable capabilities displayed by humans in making sense of an overwhelming amount of sensory information cannot be explained easily if perception is viewed as a passive process. Current theoretical and computational models assume that to achieve meaningful and coherent perception, the human brain must anticipate upcoming stimulation. But how are upcoming stimuli predicted in the brain? We unmasked the neural representation of a prediction by omitting the predicted sensory input. Electrophysiological brain signals showed that when a clear prediction can be formulated, the brain activates a template of its response to the predicted stimulus before it arrives to our senses.

Motor intention determines sensory attenuation of brain responses to self-initiated sounds.

One of the functions of the brain is to predict sensory consequences of our own actions. In auditory processing, self-initiated sounds evoke a smaller brain response than passive sound exposure of the same sound sequence. Previous work suggests that this response attenuation reflects a predictive mechanism to differentiate the sensory consequences of one's own actions from other sensory input, which seems to form the basis for the sense of agency (recognizing oneself as the agent of the movement). This study addresses the question whether attenuation of brain responses to self-initiated sounds can be explained by brain activity involved in movement planning rather than movement execution. We recorded ERPs in response to sounds initiated by button presses. In one condition, participants moved a finger to press the button voluntarily, whereas in another condition, we initiated a similar, but involuntary, finger movement by stimulating the corresponding region of the primary motor cortex with TMS. For involuntary movements, no movement intention (and no feeling of agency) could be formed; thus, no motor plans were available to the forward model. A portion of the brain response evoked by the sounds, the N1-P2 complex, was reduced in amplitude following voluntary, self-initiated movements, but not following movements initiated by motor cortex stimulation. Our findings demonstrate that movement intention and the corresponding feeling of agency determine sensory attenuation of brain responses to self-initiated sounds. The present results support the assumptions of a predictive internal forward model account operating before primary motor cortex activation.

Cortical brain states and corticospinal synchronization influence TMS-evoked motor potentials.

Transcranial magnetic stimulation (TMS) influences cortical processes. Recent findings indicate, however, that, in turn, the efficacy of TMS depends on the state of ongoing cortical oscillations. Whereas power and phase of electromyographic (EMG) activity recorded from the hand muscles as well as neural synchrony between cortex and hand muscles are known to influence the effect of TMS, to date, no study has shown an influence of the phase of cortical oscillations during wakefulness. We applied single-pulse TMS over the motor cortex and recorded motor-evoked potentials along with the electroencephalogram (EEG) and EMG. We correlated phase and power of ongoing EEG and EMG signals with the motor-evoked potential (MEP) amplitude. We also investigated the functional connectivity between cortical and hand muscle activity (corticomuscular coherence) with the MEP amplitude. EEG and EMG power and phase in a frequency band around 18 Hz correlated with the MEP amplitude. High beta-band (∼34 Hz) corticomuscular coherence exhibited a positive linear relationship with the MEP amplitude, indicating that strong synchrony between cortex and hand muscles at the moment when TMS is applied entails large MEPs. Improving upon previous studies, we demonstrate a clear dependence of TMS-induced motor effects on the state of ongoing EEG phase and power fluctuations. We conclude that not only the sampling of incoming information but also the susceptibility of cortical communication flow depends cyclically on neural phase.

Sensation of agency and perception of temporal order.

After adaptation to a fixed temporal delay between actions and their sensory consequences, stimuli delivered during the delay are perceived to occur prior to actions. Temporal judgments are also influenced by the sensation of agency (experience of causing our own actions and their sensory consequences). Sensory consequences of voluntary actions are perceived to occur earlier in time than those of involuntary actions. However, it is unclear whether temporal order illusions influence the sensation of agency. Thus, we tested how the illusionary reversal of motor actions and sound events affect the sensation of agency. We observed an absence of the sensation of agency in the auditory modality in a condition in which sounds were falsely perceived as preceding motor acts relative to the perceived temporal order in the control condition. This finding suggests a strong association between the sensation of agency and the temporal order perception of actions and their consequences.

The N1-suppression effect for self-initiated sounds is independent of attention.

BACKGROUND: If we initiate a sound by our own motor behavior, the N1 component of the auditory event-related brain potential (ERP) that the sound elicits is attenuated compared to the N1 elicited by the same sound when it is initiated externally. It has been suggested that this N1 suppression results from an internal predictive mechanism that is in the service of discriminating the sensory consequences of one's own actions from other sensory input. As the N1-suppression effect is becoming a popular approach to investigate predictive processing in cognitive and social neuroscience, it is important to exclude an alternative interpretation not related to prediction. According to the attentional account, the N1 suppression is due to a difference in the allocation of attention between self- and externally-initiated sounds. To test this hypothesis, we manipulated the allocation of attention to the sounds in different blocks: Attention was directed either to the sounds, to the own motor acts or to visual stimuli. If attention causes the N1-suppression effect, then manipulating attention should affect the effect for self-initiated sounds. RESULTS: We found N1 suppression in all conditions. The N1 per se was affected by attention, but there was no interaction between attention and self-initiation effects. This implies that self-initiation N1 effects are not caused by attention. CONCLUSIONS: The present results support the assumption that the N1-suppression effect for self-initiated sounds indicates the operation of an internal predictive mechanism. Furthermore, while attention had an influence on the N1a, N1b, and N1c components, the N1-suppression effect was confined to the N1b and N1c subcomponents suggesting that the major contribution to the auditory N1-suppression effect is circumscribed to late N1 components.

Sensory suppression and increased neuromodulation during actions disrupt memory encoding of unpredictable self-initiated stimuli.

Actions modulate sensory processing by attenuating responses to self- compared to externally generated inputs, which is traditionally attributed to stimulus-specific motor predictions. Yet, suppression has been also found for stimuli merely coinciding with actions, pointing to unspecific processes that may be driven by neuromodulatory systems. Meanwhile, the differential processing for self-generated stimuli raises the possibility of producing effects also on memory for these stimuli; however, evidence remains mixed as to the direction of the effects. Here, we assessed the effects of actions on sensory processing and memory encoding of concomitant, but unpredictable sounds, using a combination of self-generation and memory recognition task concurrently with EEG and pupil recordings. At encoding, subjects performed button presses that half of the time generated a sound (motor-auditory; MA) and listened to passively presented sounds (auditory-only; A). At retrieval, two sounds were presented and participants had to respond which one was present before. We measured memory bias and memory performance by having sequences where either both or only one of the test sounds were presented at encoding, respectively. Results showed worse memory performance - but no differences in memory bias -, attenuated responses, and larger pupil diameter for MA compared to A sounds. Critically, the larger the sensory attenuation and pupil diameter, the worse the memory performance for MA sounds. Nevertheless, sensory attenuation did not correlate with pupil dilation. Collectively, our findings suggest that sensory attenuation and neuromodulatory processes coexist during actions, and both relate to disrupted memory for concurrent, albeit unpredictable sounds.

Actions do not clearly impact auditory memory.

When memorizing a list of words, those that are read aloud are remembered better than those read silently, a phenomenon known as the production effect. There have been several attempts to understand the production effect, however, actions alone have not been examined as possible contributors. Stimuli that coincide with our own actions are processed differently compared to stimuli presented passively to us. These sensory response modulations may have an impact on how action-revolving inputs are stored in memory. In this study, we investigated whether actions could impact auditory memory. Participants listened to sounds presented either during or in between their actions. We measured electrophysiological responses to the sounds and tested participants' memory of them. Results showed attenuation of sensory responses for action-coinciding sounds. However, we did not find a significant effect on memory performance. The absence of significant behavioral findings suggests that the production effect may be not dependent on the effects of actions per se. We conclude that action alone is not sufficient to improve memory performance, and thus elicit a production effect.

Neural signatures of memory gain through active exploration in an oculomotor-auditory learning task.

Active engagement improves learning and memory, and self- versus externally generated stimuli are processed differently: perceptual intensity and neural responses are attenuated. Whether the attenuation is linked to memory formation remains unclear. This study investigates whether active oculomotor control over auditory stimuli-controlling for movement and stimulus predictability-benefits associative learning, and studies the underlying neural mechanisms. Using EEG and eye tracking we explored the impact of control during learning on the processing and memory recall of arbitrary oculomotor-auditory associations. Participants (N = 23) learned associations through active exploration or passive observation, using a gaze-controlled interface to generate sounds. Our results show faster learning progress in the active condition. ERPs time-locked to the onset of sound stimuli showed that learning progress was linked to an attenuation of the P3a component. The detection of matching movement-sound pairs triggered a target-matching P3b. There was no general modulation of ERPs through active learning. However, we found continuous variation in the strength of the memory benefit across participants: some benefited more strongly from active control during learning than others. This was paralleled in the strength of the N1 attenuation effect for self-generated stimuli, which was correlated with memory gain in active learning. Our results show that control helps learning and memory and modulates sensory responses. Individual differences during sensory processing predict the strength of the memory benefit. Taken together, these results help to disentangle the effects of agency, unspecific motor-based neuromodulation, and predictability on ERP components and establish a link between self-generation effects and active learning memory gain.

Self-generation and sound intensity interactively modulate perceptual bias, but not perceptual sensitivity.

The ability to distinguish self-generated stimuli from those caused by external sources is critical for all behaving organisms. Although many studies point to a sensory attenuation of self-generated stimuli, recent evidence suggests that motor actions can result in either attenuated or enhanced perceptual processing depending on the environmental context (i.e., stimulus intensity). The present study employed 2-AFC sound detection and loudness discrimination tasks to test whether sound source (self- or externally-generated) and stimulus intensity (supra- or near-threshold) interactively modulate detection ability and loudness perception. Self-generation did not affect detection and discrimination sensitivity (i.e., detection thresholds and Just Noticeable Difference, respectively). However, in the discrimination task, we observed a significant interaction between self-generation and intensity on perceptual bias (i.e. Point of Subjective Equality). Supra-threshold self-generated sounds were perceived softer than externally-generated ones, while at near-threshold intensities self-generated sounds were perceived louder than externally-generated ones. Our findings provide empirical support to recent theories on how predictions and signal intensity modulate perceptual processing, pointing to interactive effects of intensity and self-generation that seem to be driven by a biased estimate of perceived loudness, rather by changes in detection and discrimination sensitivity.

Standard Tone Stability as a Manipulation of Precision in the Oddball Paradigm: Modulation of Prediction Error Responses to Fixed-Probability Deviants.

Electrophysiological sensory deviance detection signals, such as the mismatch negativity (MMN), have been interpreted from the predictive coding framework as manifestations of prediction error (PE). From a frequentist perspective of the classic oddball paradigm, deviant stimuli are unexpected because of their low probability. However, the amount of PE elicited by a stimulus can be dissociated from its probability of occurrence: when the observer cannot make confident predictions, any event holds little surprise value, no matter how improbable. Here we tested the hypothesis that the magnitude of the neural response elicited to an improbable sound (D) would scale with the precision of the prediction derived from the repetition of another sound (S), by manipulating repetition stability. We recorded the Electroencephalogram (EEG) from 20 participants while passively listening to 4 types of isochronous pure tone sequences differing in the probability of the S tone (880 Hz) while holding constant the probability of the D tone [1,046 Hz; p(D) = 1/11]: Oddball [p(S) = 10/11]; High confidence (7/11); Low confidence (4/11); and Random (1/11). Tones of 9 different frequencies were equiprobably presented as fillers [p(S) + p(D) + p(F) = 1]. Using a mass-univariate non-parametric, cluster-based correlation analysis controlling for multiple comparisons, we found that the amplitude of the deviant-elicited ERP became more negative with increasing S probability, in a time-electrode window consistent with the MMN (ca. 120-200 ms; frontal), suggesting that the strength of a PE elicited to an improbable event indeed increases with the precision of the predictive model.

Attentional gain is modulated by probabilistic feature expectations in a spatial cueing task: ERP evidence.

Several theoretical and empirical studies suggest that attention and perceptual expectations influence perception in an interactive manner, whereby attentional gain is enhanced for predicted stimuli. The current study assessed whether attention and perceptual expectations interface when they are fully orthogonal, i.e., each of them relates to different stimulus features. We used a spatial cueing task with block-wise spatial attention cues that directed attention to either left or right visual field, in which Gabor gratings of either predicted (more likely) or unpredicted (less likely) orientation were presented. The lateralised posterior N1pc component was additively influenced by attention and perceptual expectations. Bayesian analysis showed no reliable evidence for the interactive effect of attention and expectations on the N1pc amplitude. However, attention and perceptual expectations interactively influenced the frontally distributed anterior N1 component (N1a). The attention effect (i.e., enhanced N1a amplitude in the attended compared to the unattended condition) was observed only for the gratings of predicted orientation, but not in the unpredicted condition. These findings suggest that attention and perceptual expectations interactively influence visual processing within 200 ms after stimulus onset and such joint influence may lead to enhanced endogenous attentional control in the dorsal fronto-parietal attention network.

Interrelation of attention and prediction in visual processing: Effects of task-relevance and stimulus probability.

The potentially interactive influence of attention and prediction was investigated by measuring event-related potentials (ERPs) in a spatial cueing task with attention (task-relevant) and prediction (probabilistic) cues. We identified distinct processing stages of this interactive influence. Firstly, in line with the attentional gain hypothesis, a larger amplitude response of the contralateral N1, and Nd1 for attended gratings was observed. Secondly, conforming to the attenuation-by-prediction hypothesis, a smaller negativity in the time window directly following the peak of the N1 component for predicted compared to unpredicted gratings was observed. In line with the hypothesis that attention and prediction interface, unpredicted/unattended stimuli elicited a larger negativity at central-parietal sites, presumably reflecting an increased prediction error signal. Thirdly, larger P3 responses to unpredicted stimuli pointed to the updating of an internal model. Attention and prediction can be considered as differentiated mechanisms that may interact at different processing stages to optimise perception.

COMT and DRD2/ANKK-1 gene-gene interaction account for resetting of gamma neural oscillations to auditory stimulus-driven attention.

Attention capture by potentially relevant environmental stimuli is critical for human survival, yet it varies considerably among individuals. A large series of studies has suggested that attention capture may depend on the cognitive balance between maintenance and manipulation of mental representations and the flexible switch between goal-directed representations and potentially relevant stimuli outside the focus of attention; a balance that seems modulated by a prefrontostriatal dopamine pathway. Here, we examined inter-individual differences in the cognitive control of attention through studying the effects of two single nucleotide polymorphisms regulating dopamine at the prefrontal cortex and the striatum (i.e., COMTMet108/158Val and ANKK1/DRD2TaqIA) on stimulus-driven attention capture. Healthy adult participants (N = 40) were assigned to different groups according to the combination of the polymorphisms COMTMet108/158Val and ANKK1/DRD2TaqIA, and were instructed to perform on a well-established distraction protocol. Performance in individuals with a balance between prefrontal dopamine display and striatal receptor density was slowed down by the occurrence of unexpected distracting events, while those with a rather unbalanced dopamine activity were able maintain task performance with no time delay, yet at the expense of a slightly lower accuracy. This advantage, associated to their distinct genetic profiles, was paralleled by an electrophysiological mechanism of phase-resetting of gamma neural oscillation to the novel, distracting events. Taken together, the current results suggest that the epistatic interaction between COMTVal108/158Met and ANKK1/DRD2 TaqIa genetic polymorphisms lies at the basis of stimulus-driven attention capture.

Sensory suppression of brain responses to self-generated sounds is observed with and without the perception of agency.

Stimuli caused by our own movements are given special treatment in the brain. Self-generated sounds evoke a smaller brain response than externally generated ones. This attenuated response may reflect a predictive mechanism to differentiate the sensory consequences of one's own actions from other sensory input. It may also relate to the feeling of being the agent of the movement and its effects, but little is known about how sensory suppression of brain responses to self-generated sounds is related to judgments of agency. To address this question, we recorded event-related potentials in response to sounds initiated by button presses. In one condition, participants perceived agency over the production of the sounds, whereas in another condition, participants experience an illusory lack of agency caused by changes in the delay between actions and effects. We compared trials in which the timing of button press and sound was physically identical, but participants' agency judgment differed. Results show reduced amplitudes of the auditory N1 component in response to self-generated sounds irrespective of agency experience, whilst P2 effects correlate with the perception of agency. Our findings suggest that suppression of the auditory N1 component to self-generated sounds does not depend on adaptation to specific action-effect time delays, and does not determine agency judgments, however, the suppression of the P2 component might relate more directly to the experience of agency.

Bridging prediction and attention in current research on perception and action.

Prediction and attention are fundamental brain functions in the service of perception and action. Theories on prediction relate to neural (mental) models inferring about (present or future) sensory or action-related information, whereas theories of attention are about the control of information flow underlying perception and action. Both concepts are related and not always clearly distinguishable. The special issue includes current research on prediction and attention in various subfields of perception and action. It especially considers interactions between predictive and attentive processes, which constitute a newly emerging and highly interesting field of research. As outlined in this editorial, the contributions in this special issue allow specifying as well as bridging concepts on prediction and attention. The joint consideration of prediction and attention also reveals common functional principles of perception and action.

Sensory suppression effects to self-initiated sounds reflect the attenuation of the unspecific N1 component of the auditory ERP.

The suppression of the auditory N1 event-related potential (ERP) to self-initiated sounds became a popular tool to tap into sensory-specific forward modeling. It is assumed that processing in the auditory cortex is attenuated due to a match between sensory stimulation and a specific sensory prediction afforded by a forward model of the motor command. The present study shows that N1 suppression was dramatically increased with long (≈ 3 s) stimulus onset asynchronies (SOA), whereas P2 suppression was equal in all SOA conditions (0.8, 1.6, 3.2 s). Thus, the P2 was found to be more sensitive to self-initiation effects than the N1 with short SOAs. Moreover, only the unspecific but not the sensory-specific N1 components were suppressed for self-initiated sounds suggesting that N1-suppression effects mainly reflect an attenuated orienting response. We argue that the N1-suppression effect is a rather indirect measure of sensory-specific forward models.

I know what is missing here: electrophysiological prediction error signals elicited by omissions of predicted "what" but not "when".

In the present study we investigated the neural code of sensory predictions. Grounded on a variety of empirical findings, we set out from the proposal that sensory predictions are coded via the top-down modulation of the sensory units whose response properties match the specific characteristics of the predicted stimulus (Albright, 2012; Arnal and Giraud, 2012). From this proposal, we derive the hypothesis that when the specific physical characteristics of the predicted stimulus cannot be advanced, the sensory system should not be able to formulate such predictions, as it would lack the means to represent them. In different conditions, participant's self-paced button presses predicted either only the precise time when a random sound would be presented (random sound condition) or both the timing and the identity of the sound (single sound condition). To isolate prediction-related activity, we inspected the event-related potential (ERP) elicited by rare omissions of the sounds following the button press (see SanMiguel et al., 2013). As expected, in the single sound condition, omissions elicited a complex response in the ERP, reflecting the presence of sound prediction and the violation of this prediction. In contrast, in the random sound condition, sound omissions were not followed by any significant responses in the ERP. These results confirmed our hypothesis, and provide support to current proposals advocating that sensory systems rely on the top-down modulation of stimulus-specific sensory representations as the neural code for prediction. In light of these findings, we discuss the significance of the omission ERP as an electrophysiological marker of predictive processing and we address the paradox that no indicators of violations of temporal prediction alone were found in the present paradigm.

Early electrophysiological indicators for predictive processing in audition: a review.

The auditory system essentially deals with sequential type of input and thus requires processing that is particularly suited to extract stimulus relations within a sequence. Evidence from a variety of paradigms converges to show that the auditory system automatically uses stimulus predictability for facilitating its sequential processing. This type of predictive processing does not require attentional processing of the sounds or cognitive control of the predictions, nor does it involve the preparation of motor responses to the auditory stimuli. We will present a taxonomy of paradigms and resulting electrophysiological indicators for such automatic predictive processing in terms of event-related potential components and oscillatory activity. These indicators will include signals of fulfilled predictions (match signals such as N1 attenuation, repetition positivity, and early evoked gamma band response enhancement) as well as signals of violated predictions (mismatch signals such as the mismatch negativity and stimulus omission responses). We will show how recent approaches have revealed particularly early indicators of predictive processing down to the level of the auditory middle-latency responses. We will discuss the strength of the various indicators in terms of a truly predictive account of auditory processing (as opposed to, e.g., a retrospective verification of predictions). Finally, we will discuss the benefits of a predictive system within and beyond auditory processing. In conclusion, we argue in favor of the overwhelming evidence for predictions in audition, flexibly instantiated on different levels and timescales, and we aim to provide guidance along a variety of research paradigms illustrating the existence of these predictions.