Unsupervised learning of perceptual feature combinations.

In many situations it is behaviorally relevant for an animal to respond to co-occurrences of perceptual, possibly polymodal features, while these features alone may have no importance. Thus, it is crucial for animals to learn such feature combinations in spite of the fact that they may occur with variable intensity and occurrence frequency. Here, we present a novel unsupervised learning mechanism that is largely independent of these contingencies and allows neurons in a network to achieve specificity for different feature combinations. This is achieved by a novel correlation-based (Hebbian) learning rule, which allows for linear weight growth and which is combined with a mechanism for gradually reducing the learning rate as soon as the neuron's response becomes feature combination specific. In a set of control experiments, we show that other existing advanced learning rules cannot satisfactorily form ordered multi-feature representations. In addition, we show that networks, which use this type of learning always stabilize and converge to subsets of neurons with different feature-combination specificity. Neurons with this property may, thus, serve as an initial stage for the processing of ecologically relevant real world situations for an animal.

Combining Optimal Path Search With Task-Dependent Learning in a Neural Network.

Finding optimal paths in connected graphs requires determining the smallest total cost for traveling along the graph's edges. This problem can be solved by several classical algorithms, where, usually, costs are predefined for all edges. Conventional planning methods can, thus, normally not be used when wanting to change costs in an adaptive way following the requirements of some task. Here, we show that one can define a neural network representation of path-finding problems by transforming cost values into synaptic weights, which allows for online weight adaptation using network learning mechanisms. When starting with an initial activity value of one, activity propagation in this network will lead to solutions, which are identical to those found by the Bellman-Ford (BF) algorithm. The neural network has the same algorithmic complexity as BF, and, in addition, we can show that network learning mechanisms (such as Hebbian learning) can adapt the weights in the network augmenting the resulting paths according to some task at hand. We demonstrate this by learning to navigate in an environment with obstacles as well as by learning to follow certain sequences of path nodes. Hence, the here-presented novel algorithm may open up a different regime of applications where path augmentation (by learning) is directly coupled with path finding in a natural way.

Simulated mental imagery for robotic task planning.

Traditional AI-planning methods for task planning in robotics require a symbolically encoded domain description. While powerful in well-defined scenarios, as well as human-interpretable, setting this up requires a substantial effort. Different from this, most everyday planning tasks are solved by humans intuitively, using mental imagery of the different planning steps. Here, we suggest that the same approach can be used for robots too, in cases which require only limited execution accuracy. In the current study, we propose a novel sub-symbolic method called Simulated Mental Imagery for Planning (SiMIP), which consists of perception, simulated action, success checking, and re-planning performed on 'imagined' images. We show that it is possible to implement mental imagery-based planning in an algorithmically sound way by combining regular convolutional neural networks and generative adversarial networks. With this method, the robot acquires the capability to use the initially existing scene to generate action plans without symbolic domain descriptions, while at the same time, plans remain human-interpretable, different from deep reinforcement learning, which is an alternative sub-symbolic approach. We create a data set from real scenes for a packing problem of having to correctly place different objects into different target slots. This way efficiency and success rate of this algorithm could be quantified.

Interpreting the decisions of CNNs via influence functions.

An understanding of deep neural network decisions is based on the interpretability of model, which provides explanations that are understandable to human beings and helps avoid biases in model predictions. This study investigates and interprets the model output based on images from the training dataset, i.e., to debug the results of a network model in relation to the training dataset. Our objective was to understand the behavior (specifically, class prediction) of deep learning models through the analysis of perturbations of the loss functions. We calculated influence scores for the VGG16 network at different hidden layers across three types of disturbances in the original images of the ImageNet dataset: texture, style, and background elimination. The global and layer-wise influence scores allowed the identification of the most influential training images for the given testing set. We illustrated our findings using influence scores by highlighting the types of disturbances that bias predictions of the network. According to our results, layer-wise influence analysis pairs well with local interpretability methods such as Shapley values to demonstrate significant differences between disturbed image subgroups. Particularly in an image classification task, our layer-wise interpretability approach plays a pivotal role to identify the classification bias in pre-trained convolutional neural networks, thus, providing useful insights to retrain specific hidden layers.

Differential Hebbian learning with time-continuous signals for active noise reduction.

Spike timing-dependent plasticity, related to differential Hebb-rules, has become a leading paradigm in neuronal learning, because weights can grow or shrink depending on the timing of pre- and post-synaptic signals. Here we use this paradigm to reduce unwanted (acoustic) noise. Our system relies on heterosynaptic differential Hebbian learning and we show that it can efficiently eliminate noise by up to -140 dB in multi-microphone setups under various conditions. The system quickly learns, most often within a few seconds, and it is robust with respect to different geometrical microphone configurations, too. Hence, this theoretical study demonstrates that it is possible to successfully transfer differential Hebbian learning, derived from the neurosciences, into a technical domain.

Finding Optimal Paths Using Networks Without Learning-Unifying Classical Approaches.

Trajectory or path planning is a fundamental issue in a wide variety of applications. In this article, we show that it is possible to solve path planning on a maze for multiple start point and endpoint highly efficiently with a novel configuration of multilayer networks that use only weighted pooling operations, for which no network training is needed. These networks create solutions, which are identical to those from classical algorithms such as breadth-first search (BFS), Dijkstra's algorithm, or TD(0). Different from competing approaches, very large mazes containing almost one billion nodes with dense obstacle configuration and several thousand importance-weighted path endpoints can this way be solved quickly in a single pass on parallel hardware.

Touching events predict human action segmentation in brain and behavior.

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information.

Predicting other people's upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person's identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action's duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects' touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions' goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

One-Shot Multi-Path Planning Using Fully Convolutional Networks in a Comparison to Other Algorithms.

Path planning plays a crucial role in many applications in robotics for example for planning an arm movement or for navigation. Most of the existing approaches to solve this problem are iterative, where a path is generated by prediction of the next state from the current state. Moreover, in case of multi-agent systems, paths are usually planned for each agent separately (decentralized approach). In case of centralized approaches, paths are computed for each agent simultaneously by solving a complex optimization problem, which does not scale well when the number of agents increases. In contrast to this, we propose a novel method, using a homogeneous, convolutional neural network, which allows generation of complete paths, even for more than one agent, in one-shot, i.e., with a single prediction step. First we consider single path planning in 2D and 3D mazes. Here, we show that our method is able to successfully generate optimal or close to optimal (in most of the cases <10% longer) paths in more than 99.5% of the cases. Next we analyze multi-paths either from a single source to multiple end-points or vice versa. Although the model has never been trained on multiple paths, it is also able to generate optimal or near-optimal (<22% longer) paths in 96.4 and 83.9% of the cases when generating two and three paths, respectively. Performance is then also compared to several state of the art algorithms.

Perceptual influence of elementary three-dimensional geometry: (2) fundamental object parts.

Objects usually consist of parts and the question arises whether there are perceptual features which allow breaking down an object into its fundamental parts without any additional (e.g., functional) information. As in the first paper of this sequence, we focus on the division of our world along convex to concave surface transitions. Here we are using machine vision to produce convex segments from 3D-scenes. We assume that a fundamental part is one, which we can easily name while at the same time there is no natural subdivision possible into smaller parts. Hence in this experiment we presented the computer vision generated segments to our participants and asked whether they can identify and name them. Additionally we control against segmentation reliability and we find a clear trend that reliable convex segments have a high degree of name-ability. In addition, we observed that using other image-segmentation methods will not yield nameable entities. This indicates that convex-concave surface transition may indeed form the basis for dividing objects into meaningful entities. It appears that other or further subdivisions do not carry such a strong semantical link to our everyday language as there are no names for them.

Perceptual influence of elementary three-dimensional geometry: (1) objectness.

Commonly complex cognitive concepts cannot consistently be connected to simple features of the world. Geometrical shape parameters and (e.g., edge features, compactness, color) may play a role for defining individual objects, but might be too variable to allow for concept formation. Earlier works had suggested that the formation of object concepts is strongly influenced by the division of our world along convex to concave surface transitions. In this first paper in a sequence of two we address this issue using abstract 3D geometrical structures (polycubes). In a first experiment, we let our subjects manipulate and compare polycubes with different compactness and different concavity/convexity asking which of them they would perceive as "an object." Both parameters (compactness and concavity/convexity) are not correlated in these stimuli. Nonetheless, we find that subjects with clear prevalence choose compact and convex ones. We continue to ask how strongly this influences the way we construct objects. Thus, in a second experiment we let humans combine polycubes to form an object. Also here we find that they prefer compact and convex configurations. This suggests that this simple geometric feature may underlie our cognitive understanding of objectness not only with respect to perception but also by influencing how we build our world.

Hippocampal place cells encode intended destination, and not a discriminative stimulus, in a conditional T-maze task.

The firing of hippocampal place cells encodes instantaneous location but can also reflect where the animal is heading (prospective firing), or where it has just come from (retrospective firing). The current experiment sought to explicitly control the prospective firing of place cells with a visual discriminada in a T-maze. Rats were trained to associate a specific visual stimulus (e.g., a flashing light) with the occurrence of reward in a specific location (e.g., the left arm of the T). A different visual stimulus (e.g., a constant light) signaled the availability of reward in the opposite arm of the T. After this discrimination had been acquired, rats were implanted with electrodes in the CA1 layer of the hippocampus. Place cells were then identified and recorded as the animals performed the discrimination task, and the presentation of the visual stimulus was manipulated. A subset of CA1 place cells fired at different rates on the central stem of the T depending on the animal's intended destination, but this conditional or prospective firing was independent of the visual discriminative stimulus. The firing rate of some place cells was, however, modulated by changes in the timing of presentation of the visual stimulus. Thus, place cells fired prospectively, but this firing did not appear to be controlled, directly, by a salient visual stimulus that controlled behavior.

Behavioral analysis of differential Hebbian learning in closed-loop systems.

Understanding closed loop behavioral systems is a non-trivial problem, especially when they change during learning. Descriptions of closed loop systems in terms of information theory date back to the 1950s, however, there have been only a few attempts which take into account learning, mostly measuring information of inputs. In this study we analyze a specific type of closed loop system by looking at the input as well as the output space. For this, we investigate simulated agents that perform differential Hebbian learning (STDP). In the first part we show that analytical solutions can be found for the temporal development of such systems for relatively simple cases. In the second part of this study we try to answer the following question: How can we predict which system from a given class would be the best for a particular scenario? This question is addressed using energy, input/output ratio and entropy measures and investigating their development during learning. This way we can show that within well-specified scenarios there are indeed agents which are optimal with respect to their structure and adaptive properties.

Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions.

Reinforcement learning methods can be used in robotics applications especially for specific target-oriented problems, for example the reward-based recalibration of goal directed actions. To this end still relatively large and continuous state-action spaces need to be efficiently handled. The goal of this paper is, thus, to develop a novel, rather simple method which uses reinforcement learning with function approximation in conjunction with different reward-strategies for solving such problems. For the testing of our method, we use a four degree-of-freedom reaching problem in 3D-space simulated by a two-joint robot arm system with two DOF each. Function approximation is based on 4D, overlapping kernels (receptive fields) and the state-action space contains about 10,000 of these. Different types of reward structures are being compared, for example, reward-on- touching-only against reward-on-approach. Furthermore, forbidden joint configurations are punished. A continuous action space is used. In spite of a rather large number of states and the continuous action space these reward/punishment strategies allow the system to find a good solution usually within about 20 trials. The efficiency of our method demonstrated in this test scenario suggests that it might be possible to use it on a real robot for problems where mixed rewards can be defined in situations where other types of learning might be difficult.

Evaluation of prognostic clinical and ECG parameters in patients after myocardial infarction by applying logistic regression method.

Noninvasive risk stratification of patients who have suffered myocardial infarction (MI) is one of the greatest challenges in today's cardiology. No single test has sufficient predictive ability. Therefore, a combination of the tests must be applied for better post-MI risk stratification. The purpose of this study was to assess noninvasive predictors of 2 years cardiac mortality in post-MI patients and create a stratification model for identification of high-risk patients. Clinical, electrocardiographic, and echocardiographic parameters were evaluated before hospital discharge in 180 survivors of acute MI (mean age 57.0 +/- 9.9, male 82.2%), followed up for 2 years. A multivariate logistic regression analysis was used to determine informative predictors of cardiac mortality. A clinical score was constructed using the regression coefficient from the multivariate model. During follow-up, 16 deaths (8.8%) occurred. Multivariate analysis identified a combination of six variables that showed the strongest association with cardiac mortality. Based on the coefficients of the logistic regression, six variables were used to create a scoring system: filtered QRS duration (QRSd) >114 ms, coefficient of variation (Cv) or=445 ms, left ventricular ejection fraction (LVEF)

Path-finding in real and simulated rats: assessing the influence of path characteristics on navigation learning.

A large body of experimental evidence suggests that the hippocampal place field system is involved in reward based navigation learning in rodents. Reinforcement learning (RL) mechanisms have been used to model this, associating the state space in an RL-algorithm to the place-field map in a rat. The convergence properties of RL-algorithms are affected by the exploration patterns of the learner. Therefore, we first analyzed the path characteristics of freely exploring rats in a test arena. We found that straight path segments with mean length 23 cm up to a maximal length of 80 cm take up a significant proportion of the total paths. Thus, rat paths are biased as compared to random exploration. Next we designed a RL system that reproduces these specific path characteristics. Our model arena is covered by overlapping, probabilistically firing place fields (PF) of realistic size and coverage. Because convergence of RL-algorithms is also influenced by the state space characteristics, different PF-sizes and densities, leading to a different degree of overlap, were also investigated. The model rat learns finding a reward opposite to its starting point. We observed that the combination of biased straight exploration, overlapping coverage and probabilistic firing will strongly impair the convergence of learning. When the degree of randomness in the exploration is increased, convergence improves, but the distribution of straight path segments becomes unrealistic and paths become 'wiggly'. To mend this situation without affecting the path characteristic two additional mechanisms are implemented: a gradual drop of the learned weights (weight decay) and path length limitation, which prevents learning if the reward is not found after some expected time. Both mechanisms limit the memory of the system and thereby counteract effects of getting trapped on a wrong path. When using these strategies individually divergent cases get substantially reduced and for some parameter settings no divergence was found anymore at all. Using weight decay and path length limitation at the same time, convergence is not much improved but instead time to convergence increases as the memory limiting effect is getting too strong. The degree of improvement relies also on the size and degree of overlap (coverage density) in the place field system. The used combination of these two parameters leads to a trade-off between convergence and speed to convergence. Thus, this study suggests that the role of the PF-system in navigation learning cannot be considered independently from the animals' exploration pattern.

Odor supported place cell model and goal navigation in rodents.

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation.

Hippocampal CA1 place cells encode intended destination on a maze with multiple choice points.

The hippocampus encodes both spatial and nonspatial aspects of a rat's ongoing behavior at the single-cell level. In this study, we examined the encoding of intended destination by hippocampal (CA1) place cells during performance of a serial reversal task on a double Y-maze. On the maze, rats had to make two choices to access one of four possible goal locations, two of which contained reward. Reward locations were kept constant within blocks of 10 trials but changed between blocks, and the session of each day comprised three or more trial blocks. A disproportionate number of place fields were observed in the start box and beginning stem of the maze, relative to other locations on the maze. Forty-six percent of these place fields had different firing rates on journeys to different goal boxes. Another group of cells had place fields before the second choice point, and, of these, 44% differentiated between journeys to specific goal boxes. In a second experiment, we observed that rats with hippocampal damage made significantly more errors than control rats on the Y-maze when reward locations were reversed. Together, these results suggest that, at the start of the maze, the hippocampus encodes both current location and the intended destination of the rat, and this encoding is necessary for the flexible response to changes in reinforcement contingencies.

Developing velocity sensitivity in a model neuron by local synaptic plasticity.

Sensor neurons, like those in the visual cortex, display specific functional properties, e.g., tuning for the orientation, direction and velocity of a moving stimulus. It is still unclear how these properties arise from the processing of the inputs which converge at a given cell. Specifically, little is known how such properties can develop by ways of synaptic plasticity. In this study we investigate the hypothesis that velocity sensitivity can develop at a neuron from different types of synaptic plasticity at different dendritic sub-structures. Specifically we are implementing spike-timing dependent plasticity at one dendritic branch and conventional long-term potentiation at another branch, both driven by dendritic spikes triggered by moving inputs. In the first part of the study, we show how velocity sensitivity can arise from such a spatially localized difference in the plasticity. In the second part we show how this scenario is augmented by the interaction between dendritic spikes and back-propagating spikes also at different dendritic branches. Recent theoretical (Saudargiene et al. in Neural Comput 16:595-626, 2004) and experimental (Froemke et al. in Nature 434:221-225, 2005) results on spatially localized plasticity suggest that such processes may play a major role in determining how synapses will change depending on their site. The current study suggests that such mechanisms could be used to develop the functional specificities of a neuron.

Self-influencing synaptic plasticity: recurrent changes of synaptic weights can lead to specific functional properties.

Recent experimental results suggest that dendritic and back-propagating spikes can influence synaptic plasticity in different ways (Holthoff, 2004; Holthoff et al., 2005). In this study we investigate how these signals could interact at dendrites in space and time leading to changing plasticity properties at local synapse clusters. Similar to a previous study (Saudargiene et al., 2004) we employ a differential Hebbian learning rule to emulate spike-timing dependent plasticity and investigate how the interaction of dendritic and back-propagating spikes, as the post-synaptic signals, could influence plasticity. Specifically, we will show that local synaptic plasticity driven by spatially confined dendritic spikes can lead to the emergence of synaptic clusters with different properties. If one of these clusters can drive the neuron into spiking, plasticity may change and the now arising global influence of a back-propagating spike can lead to a further segregation of the clusters and possibly the dying-off of some of them leading to more functional specificity. These results suggest that through plasticity being a spatial and temporal local process, the computational properties of dendrites or complete neurons can be substantially augmented.