Chaotic neural dynamics facilitate probabilistic computations through sampling.

Cortical neurons exhibit highly variable responses over trials and time. Theoretical works posit that this variability arises potentially from chaotic network dynamics of recurrently connected neurons. Here, we demonstrate that chaotic neural dynamics, formed through synaptic learning, allow networks to perform sensory cue integration in a sampling-based implementation. We show that the emergent chaotic dynamics provide neural substrates for generating samples not only of a static variable but also of a dynamical trajectory, where generic recurrent networks acquire these abilities with a biologically plausible learning rule through trial and error. Furthermore, the networks generalize their experience in the stimulus-evoked samples to the inference without partial or all sensory information, which suggests a computational role of spontaneous activity as a representation of the priors as well as a tractable biological computation for marginal distributions. These findings suggest that chaotic neural dynamics may serve for the brain function as a Bayesian generative model.

The impact of acute asymmetric hearing loss on multisensory integration.

Humans have the remarkable ability to integrate information from different senses, which greatly facilitates the detection, localization and identification of events in the environment. About 466 million people worldwide suffer from hearing loss. Yet, the impact of hearing loss on how the senses work together is rarely investigated. Here, we investigate how a common sensory impairment, asymmetric conductive hearing loss (AHL), alters the way our senses interact by examining human orienting behaviour with normal hearing (NH) and acute AHL. This type of hearing loss disrupts auditory localization. We hypothesized that this creates a conflict between auditory and visual spatial estimates and alters how auditory and visual inputs are integrated to facilitate multisensory spatial perception. We analysed the spatial and temporal properties of saccades to auditory, visual and audiovisual stimuli before and after plugging the right ear of participants. Both spatial and temporal aspects of multisensory integration were affected by AHL. Compared with NH, AHL caused participants to make slow, inaccurate and unprecise saccades towards auditory targets. Surprisingly, increased weight on visual input resulted in accurate audiovisual localization with AHL. This came at a cost: saccade latencies for audiovisual targets increased significantly. The larger the auditory localization errors, the less participants were able to benefit from audiovisual integration in terms of saccade latency. Our results indicate that observers immediately change sensory weights to effectively deal with acute AHL and preserve audiovisual accuracy in a way that cannot be fully explained by statistical models of optimal cue integration.

Common intentional binding effects across diverse sensory modalities in touch-free voluntary actions.

The intentional binding effect refers to the phenomenon where the perceived temporal interval between a voluntary action and its sensory consequence is subjectively compressed. Prior research revealed the importance of tactile feedback from the keyboard on this effect. Here we examined the necessity of such tactile feedback by utilizing a touch-free key-press device without haptic feedback, and explored how initial/outcome sensory modalities (visual/auditory/tactile) and their consistency influence the intentional binding effect. Participants estimated three delay lengths (250, 550, or 850 ms) between the initial and outcome stimuli. Results showed that regardless of the combinations of sensory modalities between the initial and the outcome stimuli (i.e., modal consistency), the intentional binding effect was only observed in the 250 ms delay condition. This findings indicate a stable intentional binding effect both within and across sensory modalities, supporting the existence of a shared mechanism underlying the binding effect in touch-free voluntary actions.

A model of cue integration as vector summation in the insect brain.

Ball-rolling dung beetles are known to integrate multiple cues in order to facilitate their straight-line orientation behaviour. Recent work has suggested that orientation cues are integrated according to a vector sum, that is, compass cues are represented by vectors and summed to give a combined orientation estimate. Further, cue weight (vector magnitude) appears to be set according to cue reliability. This is consistent with the popular Bayesian view of cue integration: cues are integrated to reduce or minimize an agent's uncertainty about the external world. Integration of orientation cues is believed to occur at the input to the insect central complex. Here, we demonstrate that a model of the head direction circuit of the central complex, including plasticity in input synapses, can act as a substrate for cue integration as vector summation. Further, we show that cue influence is not necessarily driven by cue reliability. Finally, we present a dung beetle behavioural experiment which, in combination with simulation, strongly suggests that these beetles do not weight cues according to reliability. We suggest an alternative strategy whereby cues are weighted according to relative contrast, which can also explain previous results.

Cooking through perceptual disfluencies: The effects of auditory and visual distortions on predicted and actual memory performance.

The current study investigated the joint contribution of visual and auditory disfluencies, or distortions, to actual and predicted memory performance with naturalistic, multi-modal materials through three experiments. In Experiments 1 and 2, participants watched food recipe clips containing visual and auditory information that were either fully intact or else distorted in one or both of the two modalities. They were asked to remember these for a later memory test and made memory predictions after each clip. Participants produced lower memory predictions for distorted auditory and visual information than intact ones. However, these perceptual distortions revealed no actual memory differences across encoding conditions, expanding the metacognitive illusion of perceptual disfluency for static, single-word materials to naturalistic, dynamic, multi-modal materials. Experiment 3 provided naïve participants with a hypothetical scenario about the experimental paradigm used in Experiment 1, revealing lower memory predictions for distorted than intact information in both modalities. Theoretically, these results imply that both in-the-moment experiences and a priori beliefs may contribute to the perceptual disfluency illusion. From an applied perspective, the study suggests that when audio-visual distortions occur, individuals might use this information to predict their memory performance, even when it does not factor into actual memory performance.

Taking others into account: combining directly experienced and indirect information in schizophrenia.

An abnormality in inference, resulting in distorted internal models of the world, has been argued to be a common mechanism underlying the heterogeneous psychopathology in schizophrenia. However, findings have been mixed as to wherein the abnormality lies and have typically failed to find convincing relations to symptoms. The limited and inconsistent findings may have been due to methodological limitations of the experimental design, such as conflating other factors (e.g. comprehension) with the inferential process of interest, and a failure to adequately assess and model the key aspects of the inferential process. Here, we investigated probabilistic inference based on multiple sources of information using a new digital version of the beads task, framed in a social context. Thirty-five patients with schizophrenia or schizoaffective disorder with a wide range of symptoms and 40 matched healthy control subjects performed the task, where they guessed the colour of the next marble drawn from a jar based on a sample from the jar as well as the choices and the expressed confidence of four people, each with their own independent sample (which was hidden from participant view). We relied on theoretically motivated computational models to assess which model best captured the inferential process and investigated whether it could serve as a mechanistic model for both psychotic and negative symptoms. We found that 'circular inference' best described the inference process, where patients over-weighed and overcounted direct experience and under-weighed information from others. Crucially, overcounting of direct experience was uniquely associated with most psychotic and negative symptoms. In addition, patients with worse social cognitive function had more difficulties using others' confidence to inform their choices. This difficulty was related to worse real-world functioning. The findings could not be easily ascribed to differences in working memory, executive function, intelligence or antipsychotic medication. These results suggest hallucinations, delusions and negative symptoms could stem from a common underlying abnormality in inference, where directly experienced information is assigned an unreasonable weight and taken into account multiple times. By this, even unreliable first-hand experiences may gain disproportionate significance. The effect could lead to false perceptions (hallucinations), false beliefs (delusions) and deviant social behaviour (e.g. loss of interest in others, bizarre and inappropriate behaviour). This may be particularly problematic for patients with social cognitive deficits, as they may fail to make use of corrective information from others, ultimately leading to worse social functioning.

Non-motor cues do not generate the perception of self-agency: A critique of cue-integration.

How does one know that (s)he is the causal agent of their motor actions? Earlier theories of sense of agency have attributed the capacity for perception of self-agency to the comparator process of the motor-control/action system. However, with the advent of the findings implying a role of non-motor cues (like affective states, beliefs, primed concepts, and social instructions or previews of actions) in the sense of agency literature, the perception of self-agency is hypothesized to be generated even by non-motor cues (based on their relative reliability or weighting estimate); and, this theory is come to be known as the cue-integration of sense of agency. However, the cue-integration theory motivates skepticism about whether it is falsifiable and whether it is plausible that non-motor cues that are sensorily unrelated to typical sensory processes of self-agency have the capacity to produce a perception of self-agency. To substantiate this skepticism, I critically analyze the experimental operationalizations of cue-integration-with the (classic) vicarious agency experiment as a case study-to show that (1) the participants in these experiments are ambiguous about their causal agency over motor actions, (2) thus, these participants resort to reports of self-agency as heuristic judgments (under ambiguity) rather than due to cue-integration per se, and (3) there might not have occurred cue-integration based self-agency reports if these experimental operationalizations had eliminated ambiguity about the causal agency. Thus, I conclude that the reports of self-agency (observed in typical non-motor cues based cue-integration experiments) are not instances of perceptual effect-that are hypothesized to be produced by non-motor cues-but are of heuristic judgment effect.

Integration of Motion and Form Cues for the Perception of Self-Motion in the Human Brain.

When moving around in the world, the human visual system uses both motion and form information to estimate the direction of self-motion (i.e., heading). However, little is known about cortical areas in charge of this task. This brain-imaging study addressed this question by using visual stimuli consisting of randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion [FoE]) but moved away from a different locus (the motion-defined FoE) to simulate observer translation. We first fixed the motion-defined FoE location and shifted the form-defined FoE location. We then made the locations of the motion- and the form-defined FoEs either congruent (at the same location in the display) or incongruent (on the opposite sides of the display). The motion- or the form-defined FoE shift was the same in the two types of stimuli, but the perceived heading direction shifted for the congruent, but not for the incongruent stimuli. Participants (both sexes) made a task-irrelevant (contrast discrimination) judgment during scanning. Searchlight and ROI-based multivoxel pattern analysis revealed that early visual areas V1, V2, and V3 responded to either the motion- or the form-defined FoE shift. After V3, only the dorsal areas V3a and V3B/KO responded to such shifts. Furthermore, area V3B/KO shows a significantly higher decoding accuracy for the congruent than the incongruent stimuli. Our results provide direct evidence showing that area V3B/KO does not simply respond to motion and form cues but integrates these two cues for the perception of heading.SIGNIFICANCE STATEMENT Human survival relies on accurate perception of self-motion. The visual system uses both motion (optic flow) and form cues for the perception of the direction of self-motion (heading). Although human brain areas for processing optic flow and form structure are well identified, the areas responsible for integrating these two cues for the perception of self-motion remain unknown. We conducted fMRI experiments and used multivoxel pattern analysis technique to find human brain areas that can decode the shift in heading specified by each cue alone and the two cues combined. We found that motion and form cues are first processed in the early visual areas and then are likely integrated in the higher dorsal area V3B/KO for the final estimation of heading.

Role of the pheromone for navigation in the group foraging ant, Veromessor pergandei.

Navigation is comprised of a variety of strategies which rely on multiple external cues to shape a navigator's behavioral output. Here, we explored in the ant Veromessor pergandei, the interactions between the information provided by the pheromone trail and the home vector guided by the celestial compass. We found that a cross sensory interaction between the pheromone cue and the path integrator underlies correct orientation during the inbound journey. The celestial compass provides directional information, while the presence of the trail pheromone acts as a critical context cue, triggering distinct behaviors (vector orientation, search, and backtracking). While exposed to the pheromone, foragers orient to the vector direction regardless of vector state, while in the pheromone's absence, the current remaining vector determines the forager's navigational behavior. This interaction also occurs in foragers with no remaining path integrator, relying on the activation of a celestial compass-based memory of the previous trip. Such cue interactions maximize the foragers' return to the nest and inhibit movement off the pheromone trail. Finally, our manipulations continuously rotated foragers away from their desired heading, yet foragers were proficient at counteracting these changes, steering to maintain a correct compass heading even at rotational speeds of ~ 40°/s.

Numerical averaging in mice.

Rodents can be trained to associate different durations with different stimuli (e.g., light/sound). When the associated stimuli are presented together, maximal responding is observed around the average of individual durations (akin to averaging). The current study investigated whether mice can also average independently trained numerosities. Mice were initially trained to make 10 or 20 lever presses on a single (run) lever to obtain a reward and each fixed-ratio schedule was signaled either with an auditory or visual stimulus. Then, mice were trained to press another lever to obtain the reward after they responded on the run lever for the minimum number of presses [Fixed Consecutive Number (FCN)-10 or -20 trials] signaled by the corresponding discriminative stimulus. Following this training, FCN trials with the compound stimulus were introduced to test the counting behavior of mice when they encountered conflicting information regarding the number of responses required to obtain the reward. Our results showed that the numbers of responses on these compound test trials were around the average of the number of responses in FCN-10 and FCN-20 trials particularly when the auditory stimulus was associated with a fewer number of required responses. The counting strategy explained the behavior of the majority of the mice in the FCN-Compound test trials (as opposed to the timing strategy). The number of responses in FCN-Compound trials was accounted for equally well by the arithmetic, geometric, and Bayesian averages of the number of responses observed in FCN-10 and FCN-20 trials.

Metamemory for pictures of naturalistic scenes: Assessment of accuracy and cue utilization.

Memory for naturalistic pictures is exceptionally good. However, little is known about people's ability to monitor the memorability of naturalistic pictures. We report the first systematic investigation into the accuracy and basis of metamemory in this domain. People studied pictures of naturalistic scenes, predicted their chances of recognizing each picture at a later test (judgment of learning, JOL), and completed a recognition memory test. Across three experiments, JOLs revealed substantial accuracy. This was due to people basing their JOLs on multiple cues, most of which predicted recognition memory. Identified cues include intrinsic picture attributes (e.g., peacefulness of scenes; scenes with or without persons) and extrinsic aspects of the study situation (e.g., presentation frequency; semantic distinctiveness of scenes with respect to the context). This work provides a better understanding of metamemory for pictures and it demonstrates close parallels between metamemory for naturalistic scenes and verbal materials.

A comparison of methods of assessing cue combination during navigation.

Mobile organisms make use of spatial cues to navigate effectively in the world, such as visual and self-motion cues. Over the past decade, researchers have investigated how human navigators combine spatial cues, and whether cue combination is optimal according to statistical principles, by varying the number of cues available in homing tasks. The methodological approaches employed by researchers have varied, however. One important methodological difference exists in the number of cues available to the navigator during the outbound path for single-cue trials. In some studies, navigators have access to all spatial cues on the outbound path and all but one cue is eliminated prior to execution of the return path in the single-cue conditions; in other studies, navigators only have access to one spatial cue on the outbound and return paths in the single-cue conditions. If navigators can integrate cues along the outbound path, single-cue estimates may be contaminated by the undesired cue, which will in turn affect the predictions of models of optimal cue integration. In the current experiment, we manipulated the number of cues available during the outbound path for single-cue trials, while keeping dual-cue trials constant. This variable did not affect performance in the homing task; in particular, homing performance was better in dual-cue conditions than in single-cue conditions and was statistically optimal. Both methodological approaches to measuring spatial cue integration during navigation are appropriate.

Interaction of spatial and non-spatial cues in auditory stream segregation in the European starling.

Integrating sounds from the same source and segregating sounds from different sources in an acoustic scene are an essential function of the auditory system. Naturally, the auditory system simultaneously makes use of multiple cues. Here, we investigate the interaction between spatial cues and frequency cues in stream segregation of European starlings (Sturnus vulgaris) using an objective measure of perception. Neural responses to streaming sounds were recorded, while the bird was performing a behavioural task that results in a higher sensitivity during a one-stream than a two-stream percept. Birds were trained to detect an onset time shift of a B tone in an ABA- triplet sequence in which A and B could differ in frequency and/or spatial location. If the frequency difference or spatial separation between the signal sources or both were increased, the behavioural time shift detection performance deteriorated. Spatial separation had a smaller effect on the performance compared to the frequency difference and both cues additively affected the performance. Neural responses in the primary auditory forebrain were affected by the frequency and spatial cues. However, frequency and spatial cue differences being sufficiently large to elicit behavioural effects did not reveal correlated neural response differences. The difference between the neuronal response pattern and behavioural response is discussed with relation to the task given to the bird. Perceptual effects of combining different cues in auditory scene analysis indicate that these cues are analysed independently and given different weights suggesting that the streaming percept arises consecutively to initial cue analysis.

Dynamic multimodal interactions in navigating wood ants: what do path details tell us about cue integration?

Ants are expert navigators, using multimodal information to navigate successfully. Here, we present the results of systematic studies of multimodal cue use in navigating wood ants, Formica rufa Ants learnt to navigate to a feeder that was defined by an olfactory cue (O), visual cue (V) and airflow (A) presented together. When the feeder, olfactory cue and airflow were all placed at the centre of the visual cue (VOACentre), ants did not directly approach the learnt feeder when either the olfactory or visual cue was removed. This confirms that some form of cue binding has taken place. However, in a visually simpler task with the feeder located at the edge of the visual cue (VOAEdge), ants still approached the feeder directly when individual cue components were removed. Hence, cue binding is flexible and depends on the navigational context. In general, cues act additively in determining the ants' path accuracy, i.e. the use of multiple cues increased navigation performance. Moreover, across different training conditions, we saw different motor patterns in response to different sensory cues. For instance, ants had more sinuous paths with more turns when they followed an odour plume but did not have any visual cues. Having visual information together with the odour enhanced performance and therefore positively impacted on plume following. Interestingly, path characteristics of ants from the different multimodal groups (VOACentre versus VOAEdge) were different, suggesting that the observed flexibility in cue binding may be a result of ants' movement characteristics.

Multi-modal cue integration in the black garden ant.

In a constantly changing environment, it is advantageous for animals to encode a location (such as a food source) relying on more than one single cue. A certain position might, in fact, be signalled by the presence of information acquired through different sensory modalities which may be integrated into cohesive memories. Here, we aimed to investigate multi-sensory learning capabilities and multi-modal information integration in Lasius niger ants. Individual ants were placed in a Y-maze where odour information always led to a food reward; moreover, arm and wall colour were also predictive but only when co-occurring with odour in a specific combination. At test, the odour cue was made uninformative (it was present in both arms). Ants were still able to correctly locate the reward by integrating odour with the right colour and side combination. In a second experiment, we tested whether multi-modal cue integration can take place in a single trial. To this end, ants were exposed to a rewarded odour in a single-arm maze and could experience the Y-maze (with all available cues) only once. At test (which was identical to that of Experiment 1), ants showed a slight preference for the correct colour-side combination, although not significantly different from chance level. Our results showed the capability of black garden ants to code apparently redundant contextual information and to create and rely on conditional relationships between the information available. We argue that future studies should deepen the inquiry on the timing and progression of multi-modal cue learning.

Multimodal interactions in insect navigation.

Animals travelling through the world receive input from multiple sensory modalities that could be important for the guidance of their journeys. Given the availability of a rich array of cues, from idiothetic information to input from sky compasses and visual information through to olfactory and other cues (e.g. gustatory, magnetic, anemotactic or thermal) it is no surprise to see multimodality in most aspects of navigation. In this review, we present the current knowledge of multimodal cue use during orientation and navigation in insects. Multimodal cue use is adapted to a species' sensory ecology and shapes navigation behaviour both during the learning of environmental cues and when performing complex foraging journeys. The simultaneous use of multiple cues is beneficial because it provides redundant navigational information, and in general, multimodality increases robustness, accuracy and overall foraging success. We use examples from sensorimotor behaviours in mosquitoes and flies as well as from large scale navigation in ants, bees and insects that migrate seasonally over large distances, asking at each stage how multiple cues are combined behaviourally and what insects gain from using different modalities.

Sense of agency: Sensorimotor signals and social context are differentially weighed at implicit and explicit levels.

Sense of agency (SoA) describes the experience of being the author of an action. Cue integration approaches divide SoA into an implicit level, mostly relying on prospective sensorimotor signals, and an explicit level, resulting from an integration of sensorimotor and contextual cues based on their reliability. Integration mechanisms at each level and the contribution of implicit to explicit SoA remain underspecified. In a task of movements with visual outcomes, we tested the effect of social context (contextual cue) and sensory prediction congruency (retrospective sensorimotor cue) over implicit (intentional binding) and explicit (verbal judgments) SoA. Our results suggest that prospective sensorimotor cues determine implicit SoA. At the explicit level, retrospective sensorimotor cues and contextual cues are partly integrated in an additive way, but contextual cues can also act as a heuristic if sensorimotor cues are highly unreliable. We also found no significant association between implicit and explicit SoA.

Four- and 5-year-old children adapt to the reliability of conflicting sources of information to learn novel words.

A central challenge in language acquisition is the integration of multiple sources of information, potentially in conflict, to acquire new knowledge and adjust current linguistic representations. One way to accomplish this is to assign more weight to more reliable sources of information in context. We tested the hypothesis that children adjust the weight of different sources of information during learning, considering two specific sources of information: their knowledge of the meaning of familiar words (semantics) and their familiarity with syntax. We varied the reliability of these sources of information through an induction phase (reliable syntax or reliable semantics). At test, French 4- and 5-year-old children and adults listened to sentences where information provided by these two cues conflicted and were asked to choose between two videos that illustrate the sentence. One video presented the reasonable choice if the sentence is assumed to be syntactically correct, but familiar words refer to novel things (e.g., une mange-"an eats" describes a novel object). The other video was the reasonable choice if the sentence is assumed to be syntactically incorrect and familiar words' meaning is preserved (e.g., "an eats" describes a girl eating and actually should have been "she eats"). As predicted, the proportion of syntactic choices (e.g., interpreting mange-"eats" as a novel noun) was found to be higher in the reliable syntax condition than in the reliable semantics condition, showing that children and adults can adapt their expectations to the reliability of sources of information.

Judgments of Learning in Context: Backgrounds Can Both Reduce and Produce Metamemory Illusions.

Varying item-specific features such as size (Rhodes & Castel, 2008) or blur (Yue, Castel, & Bjork, 2013) often produces metamemory illusions in which one type of item receives higher judgments of learning (JOLs) without being recalled better. In this study, we explored how similar manipulations to context would influence JOLs. When to-be-recalled words varying in size (or blur) were accompanied by backgrounds also varying in size (or blur), the traditional JOL illusions were reduced (Experiments 1, 2, 4, and 5) compared to when there were no backgrounds (Experiments 3a, 3b, and 4). Thus, the item-specific and contextual cues were used interactively. Further, the background manipulations also sometimes themselves led to metamemory illusions regarding JOLs for the to-be-remembered items. In general, there were robust individual differences in how participants used the cues, including how they incorporated the contextual cues into their JOL decisions. In part, this may explain why interactive cue utilization did not always emerge at the group level. In sum, we showed that context may affect JOLs both directly and indirectly by influencing participants' use of item-specific cues. These findings broaden our understanding of how cues may be utilized (e.g., Koriat, 1997) and integrated (e.g., Undorf, Söllner, and Bröder, 2018) in JOLs.

Border ownership-dependent tilt aftereffect for shape defined by binocular disparity and motion parallax.

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594-6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222-2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749-1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments. NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by "border ownership" neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.