Dynamic multilayer growth: Parallel vs. sequential approaches.

The decision of when to add a new hidden unit or layer is a fundamental challenge for constructive algorithms. It becomes even more complex in the context of multiple hidden layers. Growing both network width and depth offers a robust framework for leveraging the ability to capture more information from the data and model more complex representations. In the context of multiple hidden layers, should growing units occur sequentially with hidden units only being grown in one layer at a time or in parallel with hidden units growing across multiple layers simultaneously? The effects of growing sequentially or in parallel are investigated using a population dynamics-inspired growing algorithm in a multilayer context. A modified version of the constructive growing algorithm capable of growing in parallel is presented. Sequential and parallel growth methodologies are compared in a three-hidden layer multilayer perceptron on several benchmark classification tasks. Several variants of these approaches are developed for a more in-depth comparison based on the type of hidden layer initialization and the weight update methods employed. Comparisons are then made to another sequential growing approach, Dynamic Node Creation. Growing hidden layers in parallel resulted in comparable or higher performances than sequential approaches. Growing hidden layers in parallel promotes growing narrower deep architectures tailored to the task. Dynamic growth inspired by population dynamics offers the potential to grow the width and depth of deeper neural networks in either a sequential or parallel fashion.

A multilayered bidirectional associative memory model for learning nonlinear tasks.

A multilayered bidirectional associative memory neural network is proposed to account for learning nonlinear types of association. The model (denoted as the MF-BAM) is composed of two modules, the Multi-Feature extracting bidirectional associative memory (MF), which contains various unsupervised network layers, and a modified Bidirectional Associative Memory (BAM), which consists of a single supervised network layer. The MF generates successive feature patterns from the original inputs. These patterns change the relationship between the inputs and targets in a way that the BAM can learn. The model was tested on different nonlinear tasks, such as the N-bit, Double Moon and its variants, and the 3-class spiral task. Behaviors were reported through learning errors, decision zones, and recall performances. Results showed that it was possible to learn all tasks consistently. By manipulating the number of units per layer and the number of unsupervised network layers in the MF, it was possible to change the level of nonlinearity observed in the decision boundaries. Furthermore, results indicated that different behaviors were achieved from the same set of inputs by using the different generated patterns. These findings are significant as they showed how a BAM-inspired model could solve nonlinear tasks in a more cognitively plausible fashion.

Embodied working memory during ongoing input streams.

Sensory stimuli endow animals with the ability to generate an internal representation. This representation can be maintained for a certain duration in the absence of previously elicited inputs. The reliance on an internal representation rather than purely on the basis of external stimuli is a hallmark feature of higher-order functions such as working memory. Patterns of neural activity produced in response to sensory inputs can continue long after the disappearance of previous inputs. Experimental and theoretical studies have largely invested in understanding how animals faithfully maintain sensory representations during ongoing reverberations of neural activity. However, these studies have focused on preassigned protocols of stimulus presentation, leaving out by default the possibility of exploring how the content of working memory interacts with ongoing input streams. Here, we study working memory using a network of spiking neurons with dynamic synapses subject to short-term and long-term synaptic plasticity. The formal model is embodied in a physical robot as a companion approach under which neuronal activity is directly linked to motor output. The artificial agent is used as a methodological tool for studying the formation of working memory capacity. To this end, we devise a keyboard listening framework to delineate the context under which working memory content is (1) refined, (2) overwritten or (3) resisted by ongoing new input streams. Ultimately, this study takes a neurorobotic perspective to resurface the long-standing implication of working memory in flexible cognition.

Sexual affordances, perceptual-motor invariance extraction and intentional nonlinear dynamics: sexually deviant and non-deviant patterns in male subjects.

Sexual arousal and gaze behavior dynamics are used to characterize deviant sexual interests in male subjects. Pedophile patients and non-deviant subjects are immersed with virtual characters depicting relevant sexual features. Gaze behavior dynamics as indexed from correlation dimensions (D2) appears to be fractal in nature and significantly different from colored noise (surrogate data tests and recurrence plot analyses were performed). This perceptual-motor fractal dynamics parallels sexual arousal and differs from pedophiles to non-deviant subjects when critical sexual information is processed. Results are interpreted in terms of sexual affordance, perceptual invariance extraction and intentional nonlinear dynamics.

Discrimination of Motion Direction in a Robot Using a Phenomenological Model of Synaptic Plasticity.

Recognizing and tracking the direction of moving stimuli is crucial to the control of much animal behaviour. In this study, we examine whether a bio-inspired model of synaptic plasticity implemented in a robotic agent may allow the discrimination of motion direction of real-world stimuli. Starting with a well-established model of short-term synaptic plasticity (STP), we develop a microcircuit motif of spiking neurons capable of exhibiting preferential and nonpreferential responses to changes in the direction of an orientation stimulus in motion. While the robotic agent processes sensory inputs, the STP mechanism introduces direction-dependent changes in the synaptic connections of the microcircuit, resulting in a population of units that exhibit a typical cortical response property observed in primary visual cortex (V1), namely, direction selectivity. Visually evoked responses from the model are then compared to those observed in multielectrode recordings from V1 in anesthetized macaque monkeys, while sinusoidal gratings are displayed on a screen. Overall, the model highlights the role of STP as a complementary mechanism in explaining the direction selectivity and applies these insights in a physical robot as a method for validating important response characteristics observed in experimental data from V1, namely, direction selectivity.

Spiking Neurons Integrating Visual Stimuli Orientation and Direction Selectivity in a Robotic Context.

Visual motion detection is essential for the survival of many species. The phenomenon includes several spatial properties, not fully understood at the level of a neural circuit. This paper proposes a computational model of a visual motion detector that integrates direction and orientation selectivity features. A recent experiment in the Drosophila model highlights that stimulus orientation influences the neural response of direction cells. However, this interaction and the significance at the behavioral level are currently unknown. As such, another objective of this article is to study the effect of merging these two visual processes when contextualized in a neuro-robotic model and an operant conditioning procedure. In this work, the learning task was solved using an artificial spiking neural network, acting as the brain controller for virtual and physical robots, showing a behavior modulation from the integration of both visual processes.

Presence as determined by fractal perceptual-motor dynamics.

This paper presents a tentative model of the role of perceptual-motor dynamics in the emergence of the feeling of presence. A new method allowing the measure of how gaze probes three-dimensional space in immersion is used to support this model. Fractal computations of gaze behavior are shown to be more effective than standard computations of eye movements in predicting presence.

A bidirectional heteroassociative memory for binary and grey-level patterns.

Typical bidirectional associative memories (BAM) use an offline, one-shot learning rule, have poor memory storage capacity, are sensitive to noise, and are subject to spurious steady states during recall. Recent work on BAM has improved network performance in relation to noisy recall and the number of spurious attractors, but at the cost of an increase in BAM complexity. In all cases, the networks can only recall bipolar stimuli and, thus, are of limited use for grey-level pattern recall. In this paper, we introduce a new bidirectional heteroassociative memory model that uses a simple self-convergent iterative learning rule and a new nonlinear output function. As a result, the model can learn online without being subject to overlearning. Our simulation results show that this new model causes fewer spurious attractors when compared to others popular BAM networks, for a comparable performance in terms of tolerance to noise and storage capacity. In addition, the novel output function enables it to learn and recall grey-level patterns in a bidirectional way.

A sequential dynamic heteroassociative memory for multistep pattern recognition and one-to-many association.

Bidirectional associative memories (BAMs) have been widely used for auto and heteroassociative learning. However, few research efforts have addressed the issue of multistep vector pattern recognition. We propose a model that can perform multi step pattern recognition without the need for a special learning algorithm, and with the capacity to learn more than two pattern series in the training set. The model can also learn pattern series of different lengths and, contrarily to previous models, the stimuli can be composed of gray-level images. The paper also shows that by adding an extra autoassociative layer, the model can accomplish one-to-many association, a task that was exclusive to feedforward networks with context units and error backpropagation learning.

NDRAM: nonlinear dynamic recurrent associative memory for learning bipolar and nonbipolar correlated patterns.

This paper presents a new unsupervised attractor neural network, which, contrary to optimal linear associative memory models, is able to develop nonbipolar attractors as well as bipolar attractors. Moreover, the model is able to develop less spurious attractors and has a better recall performance under random noise than any other Hopfield type neural network. Those performances are obtained by a simple Hebbian/anti-Hebbian online learning rule that directly incorporates feedback from a specific nonlinear transmission rule. Several computer simulations show the model's distinguishing properties.

Scene perception, gaze behavior, and perceptual learning in virtual environments.

More and more immersive environments are developed to provide support for learning or training purposes. Ecological validity of such environments is usually based on learning performance comparisons between virtual environments and their genuine counterparts. Little is known about learning processes occurring in immersive environments. A new technique is proposed for testing perceptual learning during virtual immersion. This methodology relies upon eye-tracking technologies to analyze gaze behavior recorded in relation to virtual objects' features and tasks' requirements. It is proposed that perceptual learning mechanisms engaged could be detected through eye movements. In this study, nine subjects performed perceptual learning tasks in virtual immersion. Results obtained indicated that perceptual learning influences gaze behavior dynamics. More precisely, analysis revealed that fixation number and variability in fixation duration varied with perceptual learning level. Such findings could contribute in shedding light on learning mechanisms as well as providing additional support for validating virtual learning environments.

Unsupervised Neural Network Quantifies the Cost of Visual Information Processing.

Untrained, "flower-naïve" bumblebees display behavioural preferences when presented with visual properties such as colour, symmetry, spatial frequency and others. Two unsupervised neural networks were implemented to understand the extent to which these models capture elements of bumblebees' unlearned visual preferences towards flower-like visual properties. The computational models, which are variants of Independent Component Analysis and Feature-Extracting Bidirectional Associative Memory, use images of test-patterns that are identical to ones used in behavioural studies. Each model works by decomposing images of floral patterns into meaningful underlying factors. We reconstruct the original floral image using the components and compare the quality of the reconstructed image to the original image. Independent Component Analysis matches behavioural results substantially better across several visual properties. These results are interpreted to support a hypothesis that the temporal and energetic costs of information processing by pollinators served as a selective pressure on floral displays: flowers adapted to pollinators' cognitive constraints.

Visual choice behavior by bumblebees (Bombus impatiens) confirms unsupervised neural network's predictions.

The behavioral experiment herein tests the computational load hypothesis generated by an unsupervised neural network to examine bumblebee (Bombus impatiens) behavior at 2 visual properties: spatial frequency and symmetry. Untrained "flower-naïve" bumblebees were hypothesized to prefer symmetry only when the spatial frequency of artificial flowers is high and therefore places great information-processing demands on the bumblebees' visual system. Bumblebee choice behavior was recorded using high-definition motion-sensitive camcorders. The results support the computational model's prediction: 1-axis symmetry influenced bumblebees' preference behavior at low and high spatial frequency patterns. Additionally, increasing the level of symmetry from 1 axis to 4 axes amplified preference toward the symmetric patterns of both low and high spatial frequency patterns. The results are discussed in the context of the artificial neural network model and other hypotheses generated from the behavioral literature.