Toward understanding the neural code of the brain.

More and more, the neurosciences and the sciences concerned with mind and cognition are burying fundamental questions under layers of professional methodology. I therefore welcome Biological Cybernetics' invitation to comment on two of my papers, (von der Malsburg 1973) and (von der Malsburg and Schneider 1986) (henceforth referred to as (I) and (II)) as an opportunity to address two fundamental questions about brain and mind: How is the brain's structure generated? and How is mental content expressed by the brain's physical states? Those two questions are deeply entangled with each other and play a kind of gateway role on the way to making progress with the issues of perception, intelligence, creativity and consciousness.

Self-organization of control circuits for invariant fiber projections.

Assuming that patterns in memory are represented as two-dimensional arrays of local features, just as they are in primary visual cortices, pattern recognition can take the form of elastic graph matching (Lades et al., 1993 ). Neural implementation of this may be based on preorganized fiber projections that can be activated rapidly with the help of control units (Wolfrum, Wolff, Lücke, & von der Malsburg, 2008 ). Each control unit governs a set of projection fibers that form part of a coherent mapping. We describe a mathematical model for the ontogenesis of the underlying connectivity based on a principle of network self-organization as described by the Häussler system (Häussler & von der Malsburg, 1983 ), modified to be sensitive to pattern similarity and to support formation of multiple mappings, each under the command of a control unit. The process takes the form of a soft-winner-take-all, where units compete for the representation of maps. We show simulations for invariant point-to-point and feature-to-feature mappings.

Self-organization of topographic bilinear networks for invariant recognition.

We present a model for the emergence of ordered fiber projections that may serve as a basis for invariant recognition. After invariance transformations are self-organized, so-called control units competitively activate fiber projections for different transformation parameters. The model builds on a well-known ontogenetic mechanism, activity-based development of retinotopy, and it employs activity blobs of varying position and size to install different transformations. We provide a detailed analysis for the case of 1D input and output fields for schematic input patterns that shows how the model is able to develop specific mappings. We discuss results that show that the proposed learning scheme is stable for complex, biologically more realistic input patterns. Finally, we show that the model generalizes to 2D neuronal fields driven by simulated retinal waves.

Establishment of a scaffold for orientation maps in primary visual cortex of higher mammals.

In higher mammals, environmentally driven patterns of neural activity do not play a role in the establishment of orientation specificity and maps. It has been proposed that specific long-range interactions provide the scaffold for developing orientation maps. Our model aims at explaining how such a scaffold could develop in the first place. Broad spontaneous activity waves and locally evoked spatially periodic response pattern are used. The model is discussed in relation to biological evidence, and experiments to test the model are proposed. We show that reliable orientation specificity cannot be a result of haphazard cortical wiring, as has been proposed.

Steps toward numerical mode analysis of organizing systems.

A well established method to analyze dynamical systems described by coupled nonlinear differential equations is to determine their normal modes and reduce the dynamics, by adiabatic elimination of stable modes, to a much smaller system for the amplitudes of unstable modes and their nonlinear interactions. So far, this analysis is possible only for idealized symmetric model systems. We aim to build a framework in which realistic systems with less symmetry can be analyzed automatically. In this paper we present a first example of mode analysis with the assistance of numerical computation. Our method is illustrated using a model system for the ontogenesis of retinotopy, and the results reproduce those from theoretical analysis precisely. Aspects of organization generalized from this model system are discussed.

A recurrent dynamic model for correspondence-based face recognition.

Our aim here is to create a fully neural, functionally competitive, and correspondence-based model for invariant face recognition. By recurrently integrating information about feature similarities, spatial feature relations, and facial structure stored in memory, the system evaluates face identity ("what"-information) and face position ("where"-information) using explicit representations for both. The network consists of three functional layers of processing, (1) an input layer for image representation, (2) a middle layer for recurrent information integration, and (3) a gallery layer for memory storage. Each layer consists of cortical columns as functional building blocks that are modeled in accordance with recent experimental findings. In numerical simulations we apply the system to standard benchmark databases for face recognition. We find that recognition rates of our biologically inspired approach lie in the same range as recognition rates of recent and purely functionally motivated systems.

An applet for the Gabor similarity scaling of the differences between complex stimuli.

It is widely accepted that after the first cortical visual area, V1, a series of stages achieves a representation of complex shapes, such as faces and objects, so that they can be understood and recognized. A major challenge for the study of complex shape perception has been the lack of a principled basis for scaling of the physical differences between stimuli so that their similarity can be specified, unconfounded by early-stage differences. Without the specification of such similarities, it is difficult to make sound inferences about the contributions of later stages to neural activity or psychophysical performance. A Web-based app is described that is based on the Malsburg Gabor-jet model (Lades et al., 1993), which allows easy specification of the V1 similarity of pairs of stimuli, no matter how intricate. The model predicts the psycho physical discriminability of metrically varying faces and complex blobs almost perfectly (Yue, Biederman, Mangini, von der Malsburg, & Amir, 2012), and serves as the input stage of a large family of contemporary neurocomputational models of vision.

Computer-based recognition of dysmorphic faces.

Genetic syndromes often involve craniofacial malformations. We have investigated whether a computer can recognize disease-specific facial patterns in unrelated individuals. For this, 55 photographs (256 x 256 pixel) of patients with mucopolysaccharidosis type III (n=6), Cornelia de Lange (n=12), fragile X (n=12), Prader-Willi (n=12), and Williams-Beuren (n=13) syndromes were preprocessed by a Gabor wavelet transformation. By comparing the feature vectors at 32 facial nodes, 42/55 (76%) of the patients were correctly classified. In another four patients (7%), the correct and an incorrect diagnosis scored equally well. Clinical geneticists who were shown the same photographs achieved a recognition rate of 62%. Our results prove that certain syndromes are associated with a specific facial pattern and that this pattern can be described in mathematical terms.

Acquisition of visual shape primitives.

Shape primitives have long been proposed as components for object models in the visual system, and account for a considerable body of behavioral findings. While a large amount of effort has been devoted to the study of detection of these parts in the scenes, no research has been undertaken simulating the acquisition of these representations. We present a model which suggests how the shape primitives may be learned by experience in a self-organized fashion. This model offers the first successful unsupervised learning of shape primitives which are as complex as object parts and can serve as intermediate representations for various objects. The algorithm uses synthetic gray-level objects, each composed of several parts (primitives or else), and shape primitives emerge as a result of partial matches between several objects. Our algorithm does not use any a priori knowledge about any attributes of the patterns to be learned; and the recurrence of these visual patterns in various objects is the only basis for their emergence as new features.

The role of complex cells in object recognition.

Primate's primary visual cortex (V1) is dominated by complex cells. This choice of nature seems puzzling, as complex cells are insensitive to spatial phase--information which is generally believed to be essential for perceptual characterization and recognition of images. Modeling complex cells as Gabor wavelet magnitudes, we have mathematically and empirically examined the information content of their responses. Our results show that in spite of phase insensitivity of individual complex cell responses, population responses contain sufficient information to capture the perceptual essence of images. A complex cell type representation seems to be not only sufficiently discriminating for object identification, but also--due to its inherent ambiguities--robust to changes in background, lighting, and small deformations.

Maplets for correspondence-based object recognition.

We present a correspondence-based system for visual object recognition with invariance to position, orientation, scale and deformation. The system is intermediate between high- and low-dimensional representations of correspondences. The essence of the approach is based on higher-order links, called here maplets, which are specific to narrow ranges of mapping parameters (position, scale and orientation), which interact cooperatively with each other, and which are assumed to be formed by learning. While being based on dynamic links, the system overcomes previous problems with that formulation in terms of speed of convergence and range of allowed variation. We perform face recognition experiments, comparing ours to other published systems. We see our work as a step towards a reformulation of neural dynamics that includes rapid network self-organization as essential aspect of brain state organization.

Experience-driven formation of parts-based representations in a model of layered visual memory.

Growing neuropsychological and neurophysiological evidence suggests that the visual cortex uses parts-based representations to encode, store and retrieve relevant objects. In such a scheme, objects are represented as a set of spatially distributed local features, or parts, arranged in stereotypical fashion. To encode the local appearance and to represent the relations between the constituent parts, there has to be an appropriate memory structure formed by previous experience with visual objects. Here, we propose a model how a hierarchical memory structure supporting efficient storage and rapid recall of parts-based representations can be established by an experience-driven process of self-organization. The process is based on the collaboration of slow bidirectional synaptic plasticity and homeostatic unit activity regulation, both running at the top of fast activity dynamics with winner-take-all character modulated by an oscillatory rhythm. These neural mechanisms lay down the basis for cooperation and competition between the distributed units and their synaptic connections. Choosing human face recognition as a test task, we show that, under the condition of open-ended, unsupervised incremental learning, the system is able to form memory traces for individual faces in a parts-based fashion. On a lower memory layer the synaptic structure is developed to represent local facial features and their interrelations, while the identities of different persons are captured explicitly on a higher layer. An additional property of the resulting representations is the sparseness of both the activity during the recall and the synaptic patterns comprising the memory traces.

Figure-ground separation by cue integration.

This letter presents an improved cue integration approach to reliably separate coherent moving objects from their background scene in video sequences. The proposed method uses a probabilistic framework to unify bottom-up and top-down cues in a parallel, "democratic" fashion. The algorithm makes use of a modified Bayes rule where each pixel's posterior probabilities of figure or ground layer assignment are derived from likelihood models of three bottom-up cues and a prior model provided by a top-down cue. Each cue is treated as independent evidence for figure-ground separation. They compete with and complement each other dynamically by adjusting relative weights from frame to frame according to cue quality measured against the overall integration. At the same time, the likelihood or prior models of individual cues adapt toward the integrated result. These mechanisms enable the system to organize under the influence of visual scene structure without manual intervention. A novel contribution here is the incorporation of a top-down cue. It improves the system's robustness and accuracy and helps handle difficult and ambiguous situations, such as abrupt lighting changes or occlusion among multiple objects. Results on various video sequences are demonstrated and discussed. (Video demos are available at http://organic.usc.edu:8376/ approximately tangx/neco/index.html .).

Rapid convergence to feature layer correspondences.

We describe a neural network able to rapidly establish correspondence between neural feature layers. Each of the network's two layers consists of interconnected cortical columns, and each column consists of inhibitorily coupled subpopulations of excitatory neurons. The dynamics of the system builds on a dynamic model of a single column, which is consistent with recent experimental findings. The network realizes dynamic links between its layers with the help of specialized columns that evaluate similarities between the activity distributions of local feature cell populations, are subject to a topology constraint, and can gate the transfer of feature information between the neural layers. The system can robustly be applied to natural images, and correspondences are found in time intervals estimated to be smaller than 100 ms in physiological terms.

What is the optimal architecture for visual information routing?

Analyzing the design of networks for visual information routing is an underconstrained problem due to insufficient anatomical and physiological data. We propose here optimality criteria for the design of routing networks. For a very general architecture, we derive the number of routing layers and the fanout that minimize the required neural circuitry. The optimal fanout l is independent of network size, while the number k of layers scales logarithmically (with a prefactor below 1), with the number n of visual resolution units to be routed independently. The results are found to agree with data of the primate visual system.

Analysis of cluttered scenes using an elastic matching approach for stereo images.

We present a system for the automatic interpretation of cluttered scenes containing multiple partly occluded objects in front of unknown, complex backgrounds. The system is based on an extended elastic graph matching algorithm that allows the explicit modeling of partial occlusions. Our approach extends an earlier system in two ways. First, we use elastic graph matching in stereo image pairs to increase matching robustness and disambiguate occlusion relations. Second, we use richer feature descriptions in the object models by integrating shape and texture with color features. We demonstrate that the combination of both extensions substantially increases recognition performance. The system learns about new objects in a simple one-shot learning approach. Despite the lack of statistical information in the object models and the lack of an explicit background model, our system performs surprisingly well for this very difficult task. Our results underscore the advantages of view-based feature constellation representations for difficult object recognition problems.

Rapid processing and unsupervised learning in a model of the cortical macrocolumn.

We study a model of the cortical macrocolumn consisting of a collection of inhibitorily coupled minicolumns. The proposed system overcomes several severe deficits of systems based on single neurons as cerebral functional units, notably limited robustness to damage and unrealistically large computation time. Motivated by neuroanatomical and neurophysiological findings, the utilized dynamics is based on a simple model of a spiking neuron with refractory period, fixed random excitatory interconnection within minicolumns, and instantaneous inhibition within one macrocolumn. A stability analysis of the system's dynamical equations shows that minicolumns can act as monolithic functional units for purposes of critical, fast decisions and learning. Oscillating inhibition (in the gamma frequency range) leads to a phase-coupled population rate code and high sensitivity to small imbalances in minicolumn inputs. Minicolumns are shown to be able to organize their collective inputs without supervision by Hebbian plasticity into selective receptive field shapes, thereby becoming classifiers for input patterns. Using the bars test, we critically compare our system's performance with that of others and demonstrate its ability for distributed neural coding.

Learning the Gestalt rule of collinearity from object motion.

The Gestalt principle of collinearity (and curvilinearity) is widely regarded as being mediated by the long-range connection structure in primary visual cortex. We review the neurophysiological and psychophysical literature to argue that these connections are developed from visual experience after birth, relying on coherent object motion. We then present a neural network model that learns these connections in an unsupervised Hebbian fashion with input from real camera sequences. The model uses spatiotemporal retinal filtering, which is very sensitive to changes in the visual input. We show that it is crucial for successful learning to use the correlation of the transient responses instead of the sustained ones. As a consequence, learning works best with video sequences of moving objects. The model addresses a special case of the fundamental question of what represents the necessary a priori knowledge the brain is equipped with at birth so that the self-organized process of structuring by experience can be successful.

Image representation by complex cell responses.

We present an analysis of the representation of images as the magnitudes of their transform with complex-valued Gabor wavelets. Such a representation is a model for complex cells in the early stage of visual processing and of high technical usefulness for image understanding, because it makes the representation insensitive to small local shifts. We show that if the images are band limited and of zero mean, then reconstruction from the magnitudes is unique up to the sign for almost all images.