Emergent exploration via novelty management.

When encountering novel environments, animals perform complex yet structured exploratory behaviors. Despite their typical structuring, the principles underlying exploratory patterns are still not sufficiently understood. Here we analyzed exploratory behavioral data from two modalities: whisking and locomotion in rats and mice. We found that these rodents maximized novelty signal-to-noise ratio during each exploration episode, where novelty is defined as the accumulated information gain. We further found that these rodents maximized novelty during outbound exploration, used novelty-triggered withdrawal-like retreat behavior, and explored the environment in a novelty-descending sequence. We applied a hierarchical curiosity model, which incorporates these principles, to both modalities. We show that the model captures the major components of exploratory behavior in multiple timescales: single excursions, exploratory episodes, and developmental timeline. The model predicted that novelty is managed across exploratory modalities. Using a novel experimental setup in which mice encountered a novel object for the first time in their life, we tested and validated this prediction. Further predictions, related to the development of brain circuitry, are described. This study demonstrates that rodents select exploratory actions according to a novelty management framework and suggests a plausible mechanism by which mammalian exploration primitives can be learned during development and integrated in adult exploration of complex environments.

Learning and control of exploration primitives.

Animals explore novel environments in a cautious manner, exhibiting alternation between curiosity-driven behavior and retreats. We present a detailed formal framework for exploration behavior, which generates behavior that maintains a constant level of novelty. Similar to other types of complex behaviors, the resulting exploratory behavior is composed of exploration motor primitives. These primitives can be learned during a developmental period, wherein the agent experiences repeated interactions with environments that share common traits, thus allowing transference of motor learning to novel environments. The emergence of exploration motor primitives is the result of reinforcement learning in which information gain serves as intrinsic reward. Furthermore, actors and critics are local and ego-centric, thus enabling transference to other environments. Novelty control, i.e. the principle which governs the maintenance of constant novelty, is implemented by a central action-selection mechanism, which switches between the emergent exploration primitives and a retreat policy, based on the currently-experienced novelty. The framework has only a few parameters, wherein time-scales, learning rates and thresholds are adaptive, and can thus be easily applied to many scenarios. We implement it by modeling the rodent's whisking system and show that it can explain characteristic observed behaviors. A detailed discussion of the framework's merits and flaws, as compared to other related models, concludes the paper.

Thermodynamic control by frequent quantum measurements.

Heat flow between a large thermal 'bath' and a smaller system brings them progressively closer to thermal equilibrium while increasing their entropy. Fluctuations involving a small fraction of a statistical ensemble of systems interacting with the bath result in deviations from this trend. In this respect, quantum and classical thermodynamics are in agreement. Here we predict a different trend in a purely quantum mechanical setting: disturbances of thermal equilibrium between two-level systems (TLSs) and a bath, caused by frequent, brief quantum non-demolition measurements of the TLS energy states. By making the measurements increasingly frequent, we encounter first the anti-Zeno regime and then the Zeno regime (namely where the TLSs' relaxation respectively speeds up and slows down). The corresponding entropy and temperature of both the system and the bath are then found to either decrease or increase depending only on the rate of observation, contrary to the standard thermodynamical rules that hold for memory-less (Markov) baths. From a practical viewpoint, these anomalies may offer the possibility of very fast control of heat and entropy in quantum systems, allowing cooling and state purification over an interval much shorter than the time needed for thermal equilibration or for a feedback control loop.

Motor-sensory confluence in tactile perception.

Perception involves motor control of sensory organs. However, the dynamics underlying emergence of perception from motor-sensory interactions are not yet known. Two extreme possibilities are as follows: (1) motor and sensory signals interact within an open-loop scheme in which motor signals determine sensory sampling but are not affected by sensory processing and (2) motor and sensory signals are affected by each other within a closed-loop scheme. We studied the scheme of motor-sensory interactions in humans using a novel object localization task that enabled monitoring the relevant overt motor and sensory variables. We found that motor variables were dynamically controlled within each perceptual trial, such that they gradually converged to steady values. Training on this task resulted in improvement in perceptual acuity, which was achieved solely by changes in motor variables, without any change in the acuity of sensory readout. The within-trial dynamics is captured by a hierarchical closed-loop model in which lower loops actively maintain constant sensory coding, and higher loops maintain constant sensory update flow. These findings demonstrate interchangeability of motor and sensory variables in perception, motor convergence during perception, and a consistent hierarchical closed-loop perceptual model.

Studying Dynamics of Human Information Gathering Behaviors Using Social Robots.

A novel social interaction is a dynamic process, in which participants adapt to, react to and engage with their social partners. To facilitate such interactions, people gather information relating to the social context and structure of the situation. The current study aimed to deepen the understanding of the psychological determinants of behavior in a novel social interaction. Three social robots and the participant interacted non-verbally according to a pre-programmed "relationship matrix" that dictated who favored whom. Participants' gaze was tracked during the interaction and, using Bayesian inference models, resulted in a measure of participants' social information-gathering behaviors. Our results reveal the dynamics in a novel environment, wherein information-gathering behavior is initially predicted by psychological inflexibility and then, toward the end of the interaction, predicted by curiosity. These results highlight the utility of using social robots in behavioral experiments.

Network Dynamics of a Financial Ecosystem.

Global financial crises have led to the understanding that classical econometric models are limited in comprehending financial markets in extreme conditions, partially since they disregarded complex interactions within the system. Consequently, in recent years research efforts have been directed towards modeling the structure and dynamics of the underlying networks of financial ecosystems. However, difficulties in acquiring fine-grained empirical financial data, due to regulatory limitations, intellectual property and privacy control, still hinder the application of network analysis to financial markets. In this paper we study the trading of cryptocurrency tokens on top of the Ethereum Blockchain, which is the largest publicly available financial data source that has a granularity of individual trades and users, and which provides a rare opportunity to analyze and model financial behavior in an evolving market from its inception. This quickly developing economy is comprised of tens of thousands of different financial assets with an aggregated valuation of more than 500 Billion USD and typical daily volume of 30 Billion USD, and manifests highly volatile dynamics when viewed using classic market measures. However, by applying network theory methods we demonstrate clear structural properties and converging dynamics, indicating that this ecosystem functions as a single coherent financial market. These results suggest that a better understanding of traditional markets could become possible through the analysis of fine-grained, abundant and publicly available data of cryptomarkets.

Social behaviour as an emergent property of embodied curiosity: a robotics perspective.

Social interaction is an extremely complex yet vital component in daily life. We present a bottom-up approach for the emergence of social behaviours from the interaction of the curiosity drive, i.e. the intrinsic motivation to learn as much as possible, and the embedding environment of an agent. Implementing artificial curiosity algorithms in robots that explore human-like environments results in the emergence of a hierarchical structure of learning and behaviour. This structure resembles the sequential emergence of behavioural patterns in human babies, culminating in social behaviours, such as face detection, tracking and attention-grabbing facial expressions. These results suggest that an embodied curiosity drive may be the progenitor of many social behaviours if satiated by a social environment. This article is part of the theme issue 'From social brains to social robots: applying neurocognitive insights to human-robot interaction'.

Robot-Supported Collaborative Learning (RSCL): Social Robots as Teaching Assistants for Higher Education Small Group Facilitation.

Acknowledging the benefits of active learning and the importance of collaboration skills, the higher education system has started to transform toward utilization of group activities into lecture hall culture. In this study, a novel interaction has been introduced, wherein a social robot facilitated a small collaborative group activity of students in higher education. Thirty-six students completed a 3 h activity that covered the main content of a course in Human Computer Interaction. In this within-subject study, the students worked in groups of four on three activities, moving between three conditions: instructor facilitation of several groups using pen and paper for the activity; tablets facilitation, also used for the activity; and robot facilitation, using tablets for the activity. The robot facilitated the activity by introducing the different tasks, ensuring proper time management, and encouraging discussion among the students. This study examined the effects of facilitation type on attitudes toward the activity facilitation, the group activity, and the robot, using quantitative, and qualitative measures. Overall students perceived the robot positively, as friendly and responsive, even though the robot did not directly respond to the students' verbal communications. While most survey items did not convey significant differences between the robot, tablet, or instructor, we found significant correlations between perceptions of the robot, and attitudes toward the activity facilitation, and the group activity. Qualitative data revealed the drawbacks and benefits of the robot, as well as its relative perceived advantages over a human facilitator, such as better time management, objectivity, and efficiency. These results suggest that the robot's complementary characteristics enable a higher quality learning environment, that corresponds with students' requirements and that a Robot Supportive Collaborative Learning (RSCL) is a promising novel paradigm for higher education.

Growing Growth Mindset with a Social Robot Peer.

Mindset has been shown to have a large impact on people's academic, social, and work achievements. A growth mindset, i.e., the belief that success comes from effort and perseverance, is a better indicator of higher achievements as compared to a fixed mindset, i.e., the belief that things are set and cannot be changed. Interventions aimed at promoting a growth mindset in children range from teaching about the brain's ability to learn and change, to playing computer games that grant brain points for effort rather than success. This work explores a novel paradigm to foster a growth mindset in young children where they play a puzzle solving game with a peer-like social robot. The social robot is fully autonomous and programmed with behaviors suggestive of it having either a growth mindset or a neutral mindset as it plays puzzle games with the child. We measure the mindset of children before and after interacting with the peer-like robot, in addition to measuring their problem solving behavior when faced with a challenging puzzle. We found that children who played with a growth mindset robot 1) self-reported having a stronger growth mindset and 2) tried harder during a challenging task, as compared to children who played with the neutral mindset robot. These results suggest that interacting with peer-like social robot with a growth mindset can promote the same mindset in children.

Multi-dimensional linguistic complexity.

Linguistic complexity is a simple and elegant way of calculating complexity of strings of data. It is based on the concept that the greater the vocabulary one uses, the more complex the data. Until now, it has been used only on one-dimensional data, such as DNA and protein sequences and various human language texts. The basic definition can be extended to higher dimensions, thus allowing a practical and simple calculation of linguistic complexity of images, 3D objects and other multi-dimensional data. A simple extension of linguistic complexity is introduced, followed by 2D presentations and a discussion of parametric considerations. An example of linguistic complexity calculations, demonstrating its image processing and medical diagnostic power is presented. The subjects of this paper are patent application pending.

Reinforcement active learning in the vibrissae system: optimal object localization.

Rats move their whiskers to acquire information about their environment. It has been observed that they palpate novel objects and objects they are required to localize in space. We analyze whisker-based object localization using two complementary paradigms, namely, active learning and intrinsic-reward reinforcement learning. Active learning algorithms select the next training samples according to the hypothesized solution in order to better discriminate between correct and incorrect labels. Intrinsic-reward reinforcement learning uses prediction errors as the reward to an actor-critic design, such that behavior converges to the one that optimizes the learning process. We show that in the context of object localization, the two paradigms result in palpation whisking as their respective optimal solution. These results suggest that rats may employ principles of active learning and/or intrinsic reward in tactile exploration and can guide future research to seek the underlying neuronal mechanisms that implement them. Furthermore, these paradigms are easily transferable to biomimetic whisker-based artificial sensors and can improve the active exploration of their environment.

Tactile modulation of whisking via the brainstem loop: statechart modeling and experimental validation.

Rats repeatedly sweep their facial whiskers back and forth in order to explore their environment. Such explorative whisking appears to be driven by central pattern generators (CPGs) that operate independently of direct sensory feedback. Nevertheless, whisking can be modulated by sensory feedback, and it has been hypothesized that some of this modulation already occurs within the brainstem. However, the interaction between sensory feedback and CPG activity is poorly understood. Using the visual language of statecharts, a dynamic, bottom-up computerized model of the brainstem loop of the whisking system was built in order to investigate the interaction between sensory feedback and CPG activity during whisking behavior. As a benchmark, we used a previously quantified closed-loop phenomenon of the whisking system, touched-induced pump (TIP), which is thought to be mediated by the brainstem loop. First, we showed that TIPs depend on sensory feedback, by comparing TIP occurrence in intact rats with that in rats whose sensory nerve was experimentally cut. We then inspected several possible feedback mechanisms of TIPs using our model. The model ruled out all hypothesized mechanisms but one, which adequately simulated the corresponding motion observed in the rat. Results of the simulations suggest that TIPs are generated via sensory feedback that activates extrinsic retractor muscles in the mystacial pad. The model further predicted that in addition to the touching whisker, all whiskers found on the same side of the snout should exhibit a TIP. We present experimental results that confirm the predicted movements in behaving rats, establishing the validity of the hypothesized interaction between sensory feedback and CPG activity we suggest here for the generation of TIPs in the whisking system.

Hierarchical curiosity loops and active sensing.

A curious agent acts so as to optimize its learning about itself and its environment, without external supervision. We present a model of hierarchical curiosity loops for such an autonomous active learning agent, whereby each loop selects the optimal action that maximizes the agent's learning of sensory-motor correlations. The model is based on rewarding the learner's prediction errors in an actor-critic reinforcement learning (RL) paradigm. Hierarchy is achieved by utilizing previously learned motor-sensory mapping, which enables the learning of other mappings, thus increasing the extent and diversity of knowledge and skills. We demonstrate the relevance of this architecture to active sensing using the well-studied vibrissae (whiskers) system, where rodents acquire sensory information by virtue of repeated whisker movements. We show that hierarchical curiosity loops starting from optimally learning the internal models of whisker motion and then extending to object localization result in free-air whisking and object palpation, respectively.

Toward an integrated approach to perception and action: conference report and future directions.

This article was motivated by the conference entitled "Perception & Action - An Interdisciplinary Approach to Cognitive Systems Theory," which took place September 14-16, 2010 at the Santa Fe Institute, NM, USA. The goal of the conference was to bring together an interdisciplinary group of neuroscientists, roboticists, and theorists to discuss the extent and implications of action-perception integration in the brain. The motivation for the conference was the realization that it is a widespread approach in biological, theoretical, and computational neuroscience to investigate sensory and motor function of the brain in isolation from one another, while at the same time, it is generally appreciated that sensory and motor processing cannot be fully separated. Our article summarizes the key findings of the conference, provides a hypothetical model that integrates the major themes and concepts presented at the conference, and concludes with a perspective on future challenges in the field.

Optimal dynamical decoherence control of a qubit.

We present a theory of dynamical control by modulation for optimal decoherence reduction. The theory is based on the non-Markovian Euler-Lagrange equation for the energy-constrained field that minimizes the average dephasing rate of a qubit for any given dephasing spectrum.

A phase-space Gaussian beam summation representation of rough surface scattering.

A Gaussian beam (GB) summation representation for rough surface scattering is introduced. In this scheme, the coherent and incoherent scattered fields are described by a phase-space summation of GBs that emanate from the rough surface at discrete set of points and directions. It thus involves stochastic GB2GB scattering matrices for the coherent and incoherent fields, and deterministic GB propagators. It benefits from the simplicity and accuracy of the latter, and can be used in applications involving propagation in complex scenarios comprising inhomogeneous media with rough surface boundaries. The GB2GB matrices are calculated from the statistical moments of the scattering amplitude, which are given either analytically or empirically. An analytical and numerical example for weakly rough surface is presented and discussed. Applications to the more complicated propagation scenario of doubly rough surface waveguide with multiple reflection phenomena will be presented in a follow-up publication.

Phase-space beam summation analysis of rough surface waveguide.

A Gaussian beam summation (GBS) formulation is introduced for a doubly rough boundary waveguide, wherein the coherent and incoherent scattered fields are decomposed into a discrete phase-space summation of Gaussian beams (GB) that emanate from the rough surfaces in all directions. The scheme involves deterministic GB propagators and stochastic GB-to-GB (GB2GB) scattering matrices for the coherent and incoherent fields, where each scattered beam is propagated inside the waveguide and is scattered again from the rough boundaries. The GB2GB matrices are calculated from the statistical moments of the scattering amplitude, which are given either analytically or empirically. An analytical and numerical example for a waveguide with weak boundary roughness is presented and discussed. The formulation reveals explicitly the phase-space footprint of the stochastic multiple scattering process at the rough boundaries, thus providing a cogent physical interpretation and an effective mathematical representation to the field. The formulation also accommodates the receiver's pattern in the same phase-space format. Bistatic reverberations inside a rough surface waveguide as a function of the range and of the source and the receiver directions are thus examined as an implementation example.