Tensor Representation of Topographically Organized Semantic Spaces.

Human brains seem to represent categories of objects and actions as locations in a continuous semantic space across the cortical surface that reflects the similarity among categories. This vision of the semantic organization of information in the brain, suggested by recent experimental findings, is in harmony with the well-known topographically organized somatotopic, retinotopic, and tonotopic maps in the cerebral cortex. Here we show that these topographies can be operationally represented with context-dependent associative memories. In these models, the input vectors and, eventually also, the associated output vectors are multiplied by context vectors via the Kronecker tensor product, which allows a spatial organization of memories. Input and output tensor contexts localize matrices of semantic categories into a neural layer or slice and, at the same time, direct the flow of information arriving at the layer to a specific address, and then forward the output information toward the corresponding targets. Given a neural topographic pattern, the tensor representation will place a set of associative matrix memories within a topographic regionalized host matrix in such way that they reproduce the empirical pattern of patches in the actual neural layer. Progressive approximations to this goal are accomplished by avoiding excessive overlap of memories or the existence of empty regions within the host matrix.

Homeostasis and information processing: The key frames for the thermodynamics of biological systems.

Life is a natural phenomenon ineluctably subject to the laws and principles of physics. In this framework, thermodynamics has a crucial role, since living beings are structured on a molecular and cellular basis that can only be maintained with extensive energy consumption. This imposes that living beings are necessarily open systems. But the survival of each type of organism depends on the relative stability of certain essential variables, even in the presence of the disturbances to which they are subjected. The stability of these variables is relative in the sense that they have a narrow range of variation. This stability of the essential variables is a consequence of refined control mechanisms developed in the course of evolution, that lead to the condition called homeostasis. This homeostasis requires that control mechanisms process the various types of information related to the internal structure of the organism and its environment. Consequently, a biological system, through information processing aimed at guiding the mechanisms that maintain its homeostasis, manages the conditions imposed by the principles of thermodynamics, obtaining the most efficient use of energy possible and keeping entropic degradation controlled. In this article, we discuss the close links between thermodynamics, homeostasis and the information processing necessary to maintain homeostasis.

A memory access gate controlled by dynamic contexts.

Temporary difficulties in accessing the contents of memories are a common experience in everyday life, for example, when we try to recognize a known person in an unusual context. In addition, recent experiments seem to indicate that retrograde amnesia in the early stages of Alzheimer's disease is due to disorders in accessing memories that were installed normally. These facts suggest the existence of an intermediate step between the stimulus arrival and the associative recognition. In this work, a multimodular neurocomputational model is presented postulating the existence of a neural gate that controls the access of the stimulus with its context to the consolidated memory. If recognition is not achieved, a random search is initiated in a contextual network aroused by the initial context. The search continues until the appropriate context that allows for recognition is found or until the process is turned off because the initial stimulus is no longer maintained in the working memory. The model is based on vector patterns of neural activity and context-dependent matrix memories. Simple Markov chain simulations are presented to exemplify possible search scenarios in the contextual network. Finally, we discuss some of the characteristics of the model and the phenomenon under study.

Multiplicative processing in the modeling of cognitive activities in large neural networks.

Explaining the foundation of cognitive abilities in the processing of information by neural systems has been in the beginnings of biophysics since McCulloch and Pitts pioneered work within the biophysics school of Chicago in the 1940s and the interdisciplinary cybernetists meetings in the 1950s, inseparable from the birth of computing and artificial intelligence. Since then, neural network models have traveled a long path, both in the biophysical and the computational disciplines. The biological, neurocomputational aspect reached its representational maturity with the Distributed Associative Memory models developed in the early 70 s. In this framework, the inclusion of signal-signal multiplication within neural network models was presented as a necessity to provide matrix associative memories with adaptive, context-sensitive associations, while greatly enhancing their computational capabilities. In this review, we show that several of the most successful neural network models use a form of multiplication of signals. We present several classical models that included such kind of multiplication and the computational reasons for the inclusion. We then turn to the different proposals about the possible biophysical implementation that underlies these computational capacities. We pinpoint the important ideas put forth by different theoretical models using a tensor product representation and show that these models endow memories with the context-dependent adaptive capabilities necessary to allow for evolutionary adaptation to changing and unpredictable environments. Finally, we show how the powerful abilities of contemporary computationally deep-learning models, inspired in neural networks, also depend on multiplications, and discuss some perspectives in view of the wide panorama unfolded. The computational relevance of multiplications calls for the development of new avenues of research that uncover the mechanisms our nervous system uses to achieve multiplication.

Stabilization through self-coupling in networks of small-world and scale-free topology.

Mechanisms that ensure the stability of dynamical systems are of vital importance, in particular in our globalized and increasingly interconnected world. The so-called connectivity-stability dilemma denotes the theoretical finding that increased connectivity between the components of a large dynamical system drastically reduces its stability. This result has promoted controversies within ecology and other fields of biology, especially, because organisms as well as ecosystems constitute systems that are both highly connected and stable. Hence, it has been a major challenge to find ways to stabilize complex systems while preserving high connectivity at the same time. Investigating the stability of networks that exhibit small-world or scale-free topology is of particular interest, since these topologies have been found in many different types of real-world networks. Here, we use an approach to stabilize recurrent networks of small-world and scale-free topology by increasing the average self-coupling strength of the units of a network. For both topologies, we find that there is a sharp transition from instability to asymptotic stability. Then, most importantly, we find that the average self-coupling strength needed to stabilize a system increases much slower than its size. It appears that the qualitative shape of this relationship is the same for small-world and scale-free networks, while scale-free networks can require higher magnitudes of self-coupling. We further explore the stabilization of networks with Kronecker-Leskovec topology. Finally, we argue that our findings, in particular the stabilization of large recurrent networks through small increases in the unit self-regulation, are of practical importance for the stabilization of diverse types of complex systems.

Logic in a dynamic brain.

The ability of the human brain to carry out logical reasoning can be interpreted, in general, as a by-product of adaptive capacities of complex neural networks. Thus, we seek to base abstract logical operations in the general properties of neural networks designed as learning modules. We show that logical operations executable by McCulloch-Pitts binary networks can also be programmed in analog neural networks built with associative memory modules that process inputs as logical gates. These modules can interact among themselves to generate dynamical systems that extend the repertoire of logical operations. We demonstrate how the operations of the exclusive-OR or the implication appear as outputs of these interacting modules. In particular, we provide a model of the exclusive-OR that succeeds in evaluating an odd number of options (the exclusive-OR of classical logic fails in his case), thus paving the way for a more reasonable biological model of this important logical operator. We propose that a brain trained to compute can associate a complex logical operation to an orderly structured but temporary contingent episode by establishing a codified association among memory modules. This explanation offers an interpretation of complex logical processes (eventually learned) as associations of contingent events in memorized episodes. We suggest, as an example, a cognitive model that describes these "logical episodes".

The biological Maxwell's demons: exploring ideas about the information processing in biological systems.

This work is based on ideas supported by some of the biologists who discovered foundational facts of twentieth-century biology and who argued that Maxwell's demons are physically implemented by biological devices. In particular, JBS Haldane first, and later J. Monod, A, Lwoff and F. Jacob argued that enzymes and molecular receptors implemented Maxwell's demons that operate in systems far removed from thermodynamic equilibrium and that were responsible for creating the biological order. Later, these ideas were extended to other biological processes. In this article, we argue that these biological Maxwell's demons (BMD) are systems that have information processing capabilities that allow them to select their inputs and direct their outputs toward targets. In this context, we propose the idea that these BMD are information catalysts in which the processed information has broad thermodynamic consequences.

Schizophrenic speech as a disordered trajectory in a collapsed cognitive "Small-World".

New theoretical instruments, as goal-directed neural networks models and geometric representations based on semantic graphs, open new approaches for our understanding of the schizophrenic speech. The neuropathologic disorders of the schizophrenia can be simulated using neural models, and these models can eventually explain the origin of goal confusion and incoherence in the schizophrenic discourse trajectory. Moreover, these models are useful to evaluate the different hypothesis about the pathogenic mechanisms of the disease. At the same time, a geometric representation of the trajectory of the speech can be obtained from real data. Our conjecture is that a context-dependent graph can be constructed in order to explore if, when the disease became more severe, a transition from a quasi ordered graph to a nearly completely random graph occurs. Plausibly, there exists a wide region where the graph has the properties of a "small-world". This kind of analyses could be potentially carried out using data coming from the spontaneous speech of schizophrenic patients, and can help to evaluate the progress of the disease. At the same time, these geometrical representations could help to evaluate the effect of treatments.

The feeling of understanding: an exploration with neural models.

There exists a dynamic interaction between the world of information and the world of concepts, which is seen as a quintessential byproduct of the cultural evolution of individuals as well as of human communities. The feeling of understanding (FU) is that subjective experience that encompasses all the emotional and intellectual processes we undergo in the process of gathering evidence to achieve an understanding of an event. This experience is part of every person that has dedicated substantial efforts in scientific areas under constant research progress. The FU may have an initial growth followed by a quasi-stable regime and a possible decay when accumulated data exceeds the capacity of an individual to integrate them into an appropriate conceptual scheme. We propose a neural representation of FU based on the postulate that all cognitive activities are mapped onto dynamic neural vectors. Two models are presented that incorporate the mutual interactions among data and concepts. The first one shows how in the short time scale, FU can rise, reach a temporary steady state and subsequently decline. The second model, operating over longer scales of time, shows how a reorganization and compactification of data into global categories initiated by conceptual syntheses can yield random cycles of growth, decline and recovery of FU.

Elman topology with sigma-pi units: an application to the modeling of verbal hallucinations in schizophrenia.

The development of neural network models has greatly enhanced the comprehension of cognitive phenomena. Here, we show that models using multiplicative processing of inputs are both powerful and simple to train and understand. We believe they are valuable tools for cognitive explorations. Our model can be viewed as a subclass of networks built on sigma-pi units and we show how to derive the Kronecker product representation from the classical sigma-pi unit. We also show how the connectivity requirements of the Kronecker product can be relaxed considering statistical arguments. We use the multiplicative network to implement what we call an Elman topology, that is, a simple recurrent network (SRN) that supports aspects of language processing. As an application, we model the appearance of hallucinated voices after network damage, and show that we can reproduce results previously obtained with SRNs concerning the pathology of schizophrenia.

Modeling spatial-temporal operations with context-dependent associative memories.

We organize our behavior and store structured information with many procedures that require the coding of spatial and temporal order in specific neural modules. In the simplest cases, spatial and temporal relations are condensed in prepositions like "below" and "above", "behind" and "in front of", or "before" and "after", etc. Neural operators lie beneath these words, sharing some similarities with logical gates that compute spatial and temporal asymmetric relations. We show how these operators can be modeled by means of neural matrix memories acting on Kronecker tensor products of vectors. The complexity of these memories is further enhanced by their ability to store episodes unfolding in space and time. How does the brain scale up from the raw plasticity of contingent episodic memories to the apparent stable connectivity of large neural networks? We clarify this transition by analyzing a model that flexibly codes episodic spatial and temporal structures into contextual markers capable of linking different memory modules.

Semantic graphs and associative memories.

Graphs have been increasingly utilized in the characterization of complex networks from diverse origins, including different kinds of semantic networks. Human memories are associative and are known to support complex semantic nets; these nets are represented by graphs. However, it is not known how the brain can sustain these semantic graphs. The vision of cognitive brain activities, shown by modern functional imaging techniques, assigns renewed value to classical distributed associative memory models. Here we show that these neural network models, also known as correlation matrix memories, naturally support a graph representation of the stored semantic structure. We demonstrate that the adjacency matrix of this graph of associations is just the memory coded with the standard basis of the concept vector space, and that the spectrum of the graph is a code invariant of the memory. As long as the assumptions of the model remain valid this result provides a practical method to predict and modify the evolution of the cognitive dynamics. Also, it could provide us with a way to comprehend how individual brains that map the external reality, almost surely with different particular vector representations, are nevertheless able to communicate and share a common knowledge of the world. We finish presenting adaptive association graphs, an extension of the model that makes use of the tensor product, which provides a solution to the known problem of branching in semantic nets.

A modular approach to language production: models and facts.

Numerous cortical disorders affect language. We explore the connection between the observed language behavior and the underlying substrates by adopting a neurocomputational approach. To represent the observed trajectories of the discourse in patients with disorganized speech and in healthy participants, we design a graphical representation for the discourse as a trajectory that allows us to visualize and measure the degree of order in the discourse as a function of the disorder of the trajectories. Our work assumes that many of the properties of language production and comprehension can be understood in terms of the dynamics of modular networks of neural associative memories. Based upon this assumption, we connect three theoretical and empirical domains: (1) neural models of language processing and production, (2) statistical methods used in the construction of functional brain images, and (3) corpus linguistic tools, such as Latent Semantic Analysis (henceforth LSA), that are used to discover the topic organization of language. We show how the neurocomputational models intertwine with LSA and the mathematical basis of functional neuroimaging. Within this framework we describe the properties of a context-dependent neural model, based on matrix associative memories, that performs goal-oriented linguistic behavior. We link these matrix associative memory models with the mathematics that underlie functional neuroimaging techniques and present the "functional brain images" emerging from the model. This provides us with a completely "transparent box" with which to analyze the implication of some statistical images. Finally, we use these models to explore the possibility that functional synaptic disconnection can lead to an increase in connectivity between the representations of concepts that could explain some of the alterations in discourse displayed by patients with schizophrenia.

Detecting order-disorder transitions in discourse: implications for schizophrenia.

Several psychiatric and neurological conditions affect the semantic organization and content of a patient's speech. Specifically, the discourse of patients with schizophrenia is frequently characterized as lacking coherence. The evaluation of disturbances in discourse is often used in diagnosis and in assessing treatment efficacy, and is an important factor in prognosis. Measuring these deviations, such as "loss of meaning" and incoherence, is difficult and requires substantial human effort. Computational procedures can be employed to characterize the nature of the anomalies in discourse. We present a set of new tools derived from network theory and information science that may assist in empirical and clinical studies of communication patterns in patients, and provide the foundation for future automatic procedures. First we review information science and complex network approaches to measuring semantic coherence, and then we introduce a representation of discourse that allows for the computation of measures of disorganization. Finally we apply these tools to speech transcriptions from patients and a healthy participant, illustrating the implications and potential of this novel framework.

Dynamic searching in the brain.

Cognitive functions rely on the extensive use of information stored in the brain, and the searching for the relevant information for solving some problem is a very complex task. Human cognition largely uses biological search engines, and we assume that to study cognitive function we need to understand the way these brain search engines work. The approach we favor is to study multi-modular network models, able to solve particular problems that involve searching for information. The building blocks of these multimodular networks are the context dependent memory models we have been using for almost 20 years. These models work by associating an output to the Kronecker product of an input and a context. Input, context and output are vectors that represent cognitive variables. Our models constitute a natural extension of the traditional linear associator. We show that coding the information in vectors that are processed through association matrices, allows for a direct contact between these memory models and some procedures that are now classical in the Information Retrieval field. One essential feature of context-dependent models is that they are based on the thematic packing of information, whereby each context points to a particular set of related concepts. The thematic packing can be extended to multimodular networks involving input-output contexts, in order to accomplish more complex tasks. Contexts act as passwords that elicit the appropriate memory to deal with a query. We also show toy versions of several 'neuromimetic' devices that solve cognitive tasks as diverse as decision making or word sense disambiguation. The functioning of these multimodular networks can be described as dynamical systems at the level of cognitive variables.

Looking for robust properties in the growth of an academic network: the case of the Uruguayan biological research community.

Graph-theoretical methods have recently been used to analyze certain properties of natural and social networks. In this work, we have investigated the early stages in the growth of a Uruguayan academic network, the Biology Area of the Programme for the Development of Basic Science (PEDECIBA). This transparent social network is a territory for the exploration of the reliability of clustering methods that can potentially be used when we are confronted with opaque natural systems that provide us with a limited spectrum of observables (happens in research on the relations between brain, thought and language). From our social net, we constructed two different graph representations based on the relationships among researchers revealed by their co-participation in Master's thesis committees. We studied these networks at different times and found that they achieve connectedness early in their evolution and exhibit the small-world property (i.e. high clustering with short path lengths). The data seem compatible with power law distributions of connectivity, clustering coefficients and betweenness centrality. Evidence of preferential attachment of new nodes and of new links between old nodes was also found in both representations. These results suggest that there are topological properties observed throughout the growth of the network that do not depend on the representations we have chosen but reflect intrinsic properties of the academic collective under study. Researchers in PEDECIBA are classified according to their specialties. We analysed the community structure detected by a standard algorithm in both representations. We found that much of the pre-specified structure is recovered and part of the mismatches can be attributed to convergent interests between scientists from different sub-disciplines. This result shows the potentiality of some clustering methods for the analysis of partially known natural systems.

A hierarchical approach to cooperativity in macromolecular and self-assembling binding systems.

The study of complex macromolecular binding systems reveals that a high number of states and processes are involved in their mechanism of action, as has become more apparent with the sophistication of the experimental techniques used. The resulting information is often difficult to interpret because of the complexity of the scheme (large size and profuse interactions, including cooperative and self-assembling interactions) and the lack of transparency that this complexity introduces into the interpretation of the indexes traditionally used to describe the binding properties. In particular, cooperative behaviour can be attributed to very different causes, such as direct chemical modification of the binding sites, conformational changes in the whole structure of the macromolecule, aggregation processes between different subunits, etc. In this paper, we propose a novel approach for the analysis of the binding properties of complex macromolecular and self-assembling systems. To quantify the binding behaviour, we use the global association quotient defined as K(c) = [occupied sites]/([free sites] L), L being the free ligand concentration. K(c) can be easily related to other measures of cooperativity (such as the Hill number or the Scatchard plot) and to the free energies involved in the binding processes at each ligand concentration. In a previous work, it was shown that K(c) could be decomposed as an average of equilibrium constants in two ways: intrinsic constants for Adair binding systems and elementary constants for the general case. In this study, we show that these two decompositions are particular cases of a more general expression, where the average is over partial association quotients, associated with subsystems from which the system is composed. We also show that if the system is split into different subsystems according to a binding hierarchy that starts from the lower, microscopic level and ends at the higher, aggregation level, the global association quotient can be decomposed following the hierarchical levels of macromolecular organisation. In this process, the partial association quotients of one level are expressed, in a recursive way, as a function of the partial quotients of the level that is immediately below, until the microscopic level is reached. As a result, the binding properties of very complex macromolecular systems can be analysed in detail, making the mechanistic explanation of their behaviour transparent. In addition, our approach provides a model-independent interpretation of the intrinsic equilibrium constants in terms of the elementary ones.