Predictive coding in auditory perception: challenges and unresolved questions.

Predictive coding is arguably the currently dominant theoretical framework for the study of perception. It has been employed to explain important auditory perceptual phenomena, and it has inspired theoretical, experimental and computational modelling efforts aimed at describing how the auditory system parses the complex sound input into meaningful units (auditory scene analysis). These efforts have uncovered some vital questions, addressing which could help to further specify predictive coding and clarify some of its basic assumptions. The goal of the current review is to motivate these questions and show how unresolved issues in explaining some auditory phenomena lead to general questions of the theoretical framework. We focus on experimental and computational modelling issues related to sequential grouping in auditory scene analysis (auditory pattern detection and bistable perception), as we believe that this is the research topic where predictive coding has the highest potential for advancing our understanding. In addition to specific questions, our analysis led us to identify three more general questions that require further clarification: (1) What exactly is meant by prediction in predictive coding? (2) What governs which generative models make the predictions? and (3) What (if it exists) is the correlate of perceptual experience within the predictive coding framework?

Children's perception of visual and auditory ambiguity and its link to executive functions and creativity.

The phenomenon of perceptual bistability provides insights into aspects of perceptual processing not normally accessible to everyday experience. However, most experiments have been conducted in adults, and it is not clear to what extent key aspects of perceptual switching change through development. The current research examined the ability of 6-, 8-, and 10-year-old children (N = 66) to switch between competing percepts of ambiguous visual and auditory stimuli and links between switching rate, executive functions, and creativity. The numbers of switches participants reported in two visual tasks (ambiguous figure and ambiguous structure from motion) and two auditory tasks (verbal transformation and auditory streaming) were measured in three 60-s blocks. In addition, inhibitory control was measured with a Stroop task, set shifting was measured with a verbal fluency task, and creativity was measured with a divergent thinking task. The numbers of perceptual switches increased in all four tasks from 6 to 10 years of age but differed across tasks in that they were higher in the verbal transformation and ambigous structure-from-motion tasks than in the ambigous figure and auditory streaming tasks for all age groups. Although perceptual switching rates differed across tasks, there were predictive relationships between switching rates in some tasks. However, little evidence for the influence of central processes on perceptual switching was found. Overall, the results support the notion that perceptual switching is largely modality and task specific and that this property is already evident when perceptual switching emerges.

Relative Pitch Perception and the Detection of Deviant Tone Patterns.

Most people are able to recognise familiar tunes even when played in a different key. It is assumed that this depends on a general capacity for relative pitch perception; the ability to recognise the pattern of inter-note intervals that characterises the tune. However, when healthy adults are required to detect rare deviant melodic patterns in a sequence of randomly transposed standard patterns they perform close to chance. Musically experienced participants perform better than naïve participants, but even they find the task difficult, despite the fact that musical education includes training in interval recognition.To understand the source of this difficulty we designed an experiment to explore the relative influence of the size of within-pattern intervals and between-pattern transpositions on detecting deviant melodic patterns. We found that task difficulty increases when patterns contain large intervals (5-7 semitones) rather than small intervals (1-3 semitones). While task difficulty increases substantially when transpositions are introduced, the effect of transposition size (large vs small) is weaker. Increasing the range of permissible intervals to be used also makes the task more difficult. Furthermore, providing an initial exact repetition followed by subsequent transpositions does not improve performance. Although musical training correlates with task performance, we find no evidence that violations to musical intervals important in Western music (i.e. the perfect fifth or fourth) are more easily detected. In summary, relative pitch perception does not appear to be conducive to simple explanations based exclusively on invariant physical ratios.

Assessing the validity of subjective reports in the auditory streaming paradigm.

While subjective reports provide a direct measure of perception, their validity is not self-evident. Here, the authors tested three possible biasing effects on perceptual reports in the auditory streaming paradigm: errors due to imperfect understanding of the instructions, voluntary perceptual biasing, and susceptibility to implicit expectations. (1) Analysis of the responses to catch trials separately promoting each of the possible percepts allowed the authors to exclude participants who likely have not fully understood the instructions. (2) Explicit biasing instructions led to markedly different behavior than the conventional neutral-instruction condition, suggesting that listeners did not voluntarily bias their perception in a systematic way under the neutral instructions. Comparison with a random response condition further supported this conclusion. (3) No significant relationship was found between social desirability, a scale-based measure of susceptibility to implicit social expectations, and any of the perceptual measures extracted from the subjective reports. This suggests that listeners did not significantly bias their perceptual reports due to possible implicit expectations present in the experimental context. In sum, these results suggest that valid perceptual data can be obtained from subjective reports in the auditory streaming paradigm.

Modelling the emergence and dynamics of perceptual organisation in auditory streaming.

Many sound sources can only be recognised from the pattern of sounds they emit, and not from the individual sound events that make up their emission sequences. Auditory scene analysis addresses the difficult task of interpreting the sound world in terms of an unknown number of discrete sound sources (causes) with possibly overlapping signals, and therefore of associating each event with the appropriate source. There are potentially many different ways in which incoming events can be assigned to different causes, which means that the auditory system has to choose between them. This problem has been studied for many years using the auditory streaming paradigm, and recently it has become apparent that instead of making one fixed perceptual decision, given sufficient time, auditory perception switches back and forth between the alternatives-a phenomenon known as perceptual bi- or multi-stability. We propose a new model of auditory scene analysis at the core of which is a process that seeks to discover predictable patterns in the ongoing sound sequence. Representations of predictable fragments are created on the fly, and are maintained, strengthened or weakened on the basis of their predictive success, and conflict with other representations. Auditory perceptual organisation emerges spontaneously from the nature of the competition between these representations. We present detailed comparisons between the model simulations and data from an auditory streaming experiment, and show that the model accounts for many important findings, including: the emergence of, and switching between, alternative organisations; the influence of stimulus parameters on perceptual dominance, switching rate and perceptual phase durations; and the build-up of auditory streaming. The principal contribution of the model is to show that a two-stage process of pattern discovery and competition between incompatible patterns can account for both the contents (perceptual organisations) and the dynamics of human perception in auditory streaming.

Predictive regularity representations in violation detection and auditory stream segregation: from conceptual to computational models.

Predictive accounts of perception have received increasing attention in the past 20 years. Detecting violations of auditory regularities, as reflected by the Mismatch Negativity (MMN) auditory event-related potential, is amongst the phenomena seamlessly fitting this approach. Largely based on the MMN literature, we propose a psychological conceptual framework called the Auditory Event Representation System (AERS), which is based on the assumption that auditory regularity violation detection and the formation of auditory perceptual objects are based on the same predictive regularity representations. Based on this notion, a computational model of auditory stream segregation, called CHAINS, has been developed. In CHAINS, the auditory sensory event representation of each incoming sound is considered for being the continuation of likely combinations of the preceding sounds in the sequence, thus providing alternative interpretations of the auditory input. Detecting repeating patterns allows predicting upcoming sound events, thus providing a test and potential support for the corresponding interpretation. Alternative interpretations continuously compete for perceptual dominance. In this paper, we briefly describe AERS and deduce some general constraints from this conceptual model. We then go on to illustrate how these constraints are computationally specified in CHAINS.

The effects of rhythm and melody on auditory stream segregation.

While many studies have assessed the efficacy of similarity-based cues for auditory stream segregation, much less is known about whether and how the larger-scale structure of sound sequences support stream formation and the choice of sound organization. Two experiments investigated the effects of musical melody and rhythm on the segregation of two interleaved tone sequences. The two sets of tones fully overlapped in pitch range but differed from each other in interaural time and intensity. Unbeknownst to the listener, separately, each of the interleaved sequences was created from the notes of a different song. In different experimental conditions, the notes and/or their timing could either follow those of the songs or they could be scrambled or, in case of timing, set to be isochronous. Listeners were asked to continuously report whether they heard a single coherent sequence (integrated) or two concurrent streams (segregated). Although temporal overlap between tones from the two streams proved to be the strongest cue for stream segregation, significant effects of tonality and familiarity with the songs were also observed. These results suggest that the regular temporal patterns are utilized as cues in auditory stream segregation and that long-term memory is involved in this process.

A neurocomputational model of stimulus-specific adaptation to oddball and Markov sequences.

Stimulus-specific adaptation (SSA) occurs when the spike rate of a neuron decreases with repetitions of the same stimulus, but recovers when a different stimulus is presented. It has been suggested that SSA in single auditory neurons may provide information to change detection mechanisms evident at other scales (e.g., mismatch negativity in the event related potential), and participate in the control of attention and the formation of auditory streams. This article presents a spiking-neuron model that accounts for SSA in terms of the convergence of depressing synapses that convey feature-specific inputs. The model is anatomically plausible, comprising just a few homogeneously connected populations, and does not require organised feature maps. The model is calibrated to match the SSA measured in the cortex of the awake rat, as reported in one study. The effect of frequency separation, deviant probability, repetition rate and duration upon SSA are investigated. With the same parameter set, the model generates responses consistent with a wide range of published data obtained in other auditory regions using other stimulus configurations, such as block, sequential and random stimuli. A new stimulus paradigm is introduced, which generalises the oddball concept to Markov chains, allowing the experimenter to vary the tone probabilities and the rate of switching independently. The model predicts greater SSA for higher rates of switching. Finally, the issue of whether rarity or novelty elicits SSA is addressed by comparing the responses of the model to deviants in the context of a sequence of a single standard or many standards. The results support the view that synaptic adaptation alone can explain almost all aspects of SSA reported to date, including its purported novelty component, and that non-trivial networks of depressing synapses can intensify this novelty response.

Abstract stimulus-specific adaptation models.

Many neurons that initially respond to a stimulus stop responding if the stimulus is presented repeatedly but recover their response if a different stimulus is presented. This phenomenon is referred to as stimulus-specific adaptation (SSA). SSA has been investigated extensively using oddball experiments, which measure the responses of a neuron to sequences of stimuli. Neurons that exhibit SSA respond less vigorously to common stimuli, and the metric typically used to quantify this difference is the SSA index (SI). This article presents the first detailed analysis of the SI metric by examining the question: How should a system (e.g., a neuron) respond to stochastic input if it is to maximize the SI of its output? Questions like this one are particularly relevant to those wishing to construct computational models of SSA. If an artificial neural network receives stimulus information at a particular rate and must respond within a fixed time, what is the highest SI one can reasonably expect? We demonstrate that the optimum, average SI is constrained by the information in the input source, the length and encoding of the memory, and the assumptions concerning how the task is decomposed.

The role of feedback in a hierarchical model of object perception.

We present a model which stems from a well-established model of object recognition, HMAX, and show how this feedforward system can include feedback, using a recently proposed architecture which reconciles biased competition and predictive coding approaches. Simulation results show successful feedforward object recognition, including cases of occluded and illusory images. Recognition is both position and size invariant. The model also provides a functional interpretation of the role of feedback connectivity in accounting for several observed effects such as enhancement, suppression and refinement of activity in lower areas. The model can qualitatively replicate responses in early visual cortex to occluded and illusory contours; and fMRI data showing that high-level object recognition reduces activity in lower areas. A Gestalt-like mechanism based on collinearity, co-orientation and good continuation principles is proposed to explain illusory contour formation which allows the system to adapt a single high-level object prototype to illusory Kanizsa figures of different sizes, shapes and positions. Overall the model provides a biophysiologically plausible interpretation, supported by current experimental evidence, of the interaction between top-down global feedback and bottom-up local evidence in the context of hierarchical object perception.

Emergent feature sensitivity in a model of the auditory thalamocortical system.

If, as is widely believed, perception is based upon the responses of neurons that are tuned to stimulus features, then precisely what features are encoded and how do neurons in the system come to be sensitive to those features? Here we show differential responses to ripple stimuli can arise through exposure to formative stimuli in a recurrently connected model of the thalamocortical system which exhibits delays, lateral and recurrent connections, and learning in the form of spike timing dependent plasticity.

STDP pattern onset learning depends on background activity.

Spike-timing dependent plasticity is a learning mechanism used extensively within neural modelling. The learning rule has previously been shown to allow a neuron to learn a repeated spatio-temporal pattern among its afferents and respond at its onset. In this study we reconfirm these previous results and additionally adduce that such learning is dependent on background activity. Furthermore, we found that the onset learning is unstable when in a noisy framework. Specifically, if the level of background activity changes during learning the response latency of a neuron may increase and with the adding of additional noise the distribution of response latencies degrades. Consequently, we present preliminary insights into the neuron's encoding: viz. that a neuron may encode the coincidence of spikes from a subsection of a stimulus' afferents, but the temporal precision of the onset response depends on some background activity, which must be similar to that present during learning.

Understanding pitch perception as a hierarchical process with top-down modulation.

Pitch is one of the most important features of natural sounds, underlying the perception of melody in music and prosody in speech. However, the temporal dynamics of pitch processing are still poorly understood. Previous studies suggest that the auditory system uses a wide range of time scales to integrate pitch-related information and that the effective integration time is both task- and stimulus-dependent. None of the existing models of pitch processing can account for such task- and stimulus-dependent variations in processing time scales. This study presents an idealized neurocomputational model, which provides a unified account of the multiple time scales observed in pitch perception. The model is evaluated using a range of perceptual studies, which have not previously been accounted for by a single model, and new results from a neurophysiological experiment. In contrast to other approaches, the current model contains a hierarchy of integration stages and uses feedback to adapt the effective time scales of processing at each stage in response to changes in the input stimulus. The model has features in common with a hierarchical generative process and suggests a key role for efferent connections from central to sub-cortical areas in controlling the temporal dynamics of pitch processing.

Regular patterns stabilize auditory streams.

The auditory system continuously parses the acoustic environment into auditory objects, usually representing separate sound sources. Sound sources typically show characteristic emission patterns. These regular temporal sound patterns are possible cues for distinguishing sound sources. The present study was designed to test whether regular patterns are used as cues for source distinction and to specify the role that detecting these regularities may play in the process of auditory stream segregation. Participants were presented with tone sequences, and they were asked to continuously indicate whether they perceived the tones in terms of a single coherent sequence of sounds (integrated) or as two concurrent sound streams (segregated). Unknown to the participant, in some stimulus conditions, regular patterns were present in one or both putative streams. In all stimulus conditions, participants' perception switched back and forth between the two sound organizations. Importantly, regular patterns occurring in either one or both streams prolonged the mean duration of two-stream percepts, whereas the duration of one-stream percepts was unaffected. These results suggest that temporal regularities are utilized in auditory scene analysis. It appears that the role of this cue lies in stabilizing streams once they have been formed on the basis of simpler acoustic cues.

A cascade autocorrelation model of pitch perception.

Autocorrelation algorithms, in combination with computational models of the auditory periphery, have been successfully used to predict the pitch of a wide range of complex stimuli. However, new stimuli are frequently offered as counterexamples to the viability of this approach. This study addresses the issue of whether in the light of these challenges the predictive power of autocorrelation can be preserved by changes to the peripheral model and the computational algorithm. An existing model is extended by the addition of a low-pass filter of the summary integration of the individual within-channel autocorrelations. Other recent developments are also incorporated, including nonlinear processing on the basilar membrane and the use of integration time constants that are proportional to the autocorrelation lags. The modified and extended model predicts with reasonable success the pitches of a range of stimuli that have proved problematic for earlier implementations of the autocorrelation principle. The evaluation stimuli include short tone sequences, click trains consisting of alternating interclick intervals, click trains consisting of mixtures of regular and irregular intervals, shuffled click trains, and transposed tones.

Functional brain networks underlying idiosyncratic switching patterns in multi-stable auditory perception.

In perceptual multi-stability, perception stochastically switches between alternative interpretations of the stimulus allowing examination of perceptual experience independent of stimulus parameters. Previous studies found that listeners show temporally stable idiosyncratic switching patterns when listening to a multi-stable auditory stimulus, such as in the auditory streaming paradigm. This inter-individual variability can be described along two dimensions, Exploration and Segregation. In the current study, we explored the functional brain networks associated with these dimensions and their constituents using electroencephalography. Results showed that Segregation and its constituents are related to brain networks operating in the theta EEG band, whereas Exploration and its constituents are related to networks in the lower and upper alpha and beta bands. Thus, the dimensions on which individuals' perception differ from each other in the auditory streaming paradigm probably reflect separate perceptual processes in the human brain. Further, the results suggest that networks mainly located in left auditory areas underlie the perception of integration, whereas perceiving the alternative patterns is accompanied by stronger interhemispheric connections.

A test of the adaptive network explanation of functional disorders using a machine learning analysis of symptoms.

The classification and etiology of functional disorders is controversial. Evidence supports both psychological and biological (disease) models that show, respectively, that functional disorders should be classified as one (bodily distress syndrome) and many (e.g., irritable bowel syndrome (IBS), fibromyalgia syndrome (FMS), and chronic fatigue syndrome (CFS)). Two network models (symptom network and adaptive network) can explain the specificity and covariation of symptomatology, but only the adaptive network model can explain the covariation of the somatic symptoms of functional disorders. The adaptive network model is based on the premise that a network of biological mechanisms has emergent properties and can exhibit adaptation. The purpose of this study was to test the predictions that symptom similarity increases with pathology and that network connection strengths vary with pathology, as this would be consistent with the notion that functional disorder pathology arises from network adaptation. We conducted a symptom internet survey followed by machine learning analysis. Participants were 1751 people reporting IBS, FMS or CFS diagnosis who completed a 61-item symptom questionnaire. Eleven symptom clusters were identified. Differences in symptom clusters between IBS, FMS and CFS groups decreased as overall symptom frequency increased. The strength of outgoing connections between clusters varied as a function of symptom frequency and single versus multiple diagnoses. The findings suggest that the pathology of functional disorders involves an increase in the activity and causal connections between several symptom causing mechanisms. The data provide support for the proposal that the body is capable of complex adaptation and that functional disorders result when rules that normally improve adaptation create maladaptive change.

Similar but separate systems underlie perceptual bistability in vision and audition.

The dynamics of perceptual bistability, the phenomenon in which perception switches between different interpretations of an unchanging stimulus, are characterised by very similar properties across a wide range of qualitatively different paradigms. This suggests that perceptual switching may be triggered by some common source. However, it is also possible that perceptual switching may arise from a distributed system, whose components vary according to the specifics of the perceptual experiences involved. Here we used a visual and an auditory task to determine whether individuals show cross-modal commonalities in perceptual switching. We found that individual perceptual switching rates were significantly correlated across modalities. We then asked whether perceptual switching arises from some central (modality-) task-independent process or from a more distributed task-specific system. We found that a log-normal distribution best explained the distribution of perceptual phases in both modalities, suggestive of a combined set of independent processes causing perceptual switching. Modality- and/or task-dependent differences in these distributions, and lack of correlation with the modality-independent central factors tested (ego-resiliency, creativity, and executive function), also point towards perceptual switching arising from a distributed system of similar but independent processes.

The role of transients in auditory processing.

Enhancement of auditory transients is well documented in the auditory periphery and mid-brain, and single unit investigations have identified units with responses which may underlie this sensitivity. It is also known that transients are important in psychophysics in, for example, speech comprehension and object recognition and grouping. In this work we use a simple phenomenological model of auditory transient extraction, based on the skewness of the distribution of energy inside a frequency dependent time window, and show that this view is consistent with electrophysiological measurements of auditory brainstem responses. In addition, we present evidence that this representation may provide a positive biological advantage in processing classes of sound that are behaviourlly relevant.

The Risky Side of Creativity: Domain Specific Risk Taking in Creative Individuals.

Risk taking is often associated with creativity, yet little evidence exists to support this association. The present article aimed to systematically explore this association. In two studies, we investigated the relationship between five different domains of risk taking (financial, health and safety, recreational, ethical and social) and five different measures of creativity. Results from the first (laboratory-based) offline study suggested that creativity is associated with high risk taking tendencies in the social domain but not the other domains. Indeed, in the second study conducted online with a larger and diverse sample, the likelihood of social risk taking was the strongest predictor of creative personality and ideation scores. These findings illustrate the necessity to treat creativity and risk taking as multi-dimensional traits and the need to have a more nuanced framework of creativity and other related cognitive functions.