Retrieval capabilities of hierarchical networks: from Dyson to Hopfield.

We consider statistical-mechanics models for spin systems built on hierarchical structures, which provide a simple example of non-mean-field framework. We show that the coupling decay with spin distance can give rise to peculiar features and phase diagrams much richer than their mean-field counterpart. In particular, we consider the Dyson model, mimicking ferromagnetism in lattices, and we prove the existence of a number of metastabilities, beyond the ordered state, which become stable in the thermodynamic limit. Such a feature is retained when the hierarchical structure is coupled with the Hebb rule for learning, hence mimicking the modular architecture of neurons, and gives rise to an associative network able to perform single pattern retrieval as well as multiple-pattern retrieval, depending crucially on the external stimuli and on the rate of interaction decay with distance; however, those emergent multitasking features reduce the network capacity with respect to the mean-field counterpart. The analysis is accomplished through statistical mechanics, Markov chain theory, signal-to-noise ratio technique, and numerical simulations in full consistency. Our results shed light on the biological complexity shown by real networks, and suggest future directions for understanding more realistic models.

Anergy in self-directed B lymphocytes: A statistical mechanics perspective.

Self-directed lymphocytes may evade clonal deletion at ontogenesis but still remain harmless due to a mechanism called clonal anergy. For B-lymphocytes, two major explanations for anergy developed over the last decades: according to Varela theory, anergy stems from a proper orchestration of the whole B-repertoire, such that self-reactive clones, due to intensive feed-back from other clones, display strong inertia when mounting a response. Conversely, according to the model of cognate response, self-reacting cells are not stimulated by helper lymphocytes and the absence of such signaling yields anergy. Through statistical mechanics we show that helpers do not prompt activation of a sub-group of B-cells: remarkably, the latter are just those broadly interacting in the idiotypic network. Hence Varela theory can finally be reabsorbed into the prevailing framework of the cognate response model. Further, we show how the B-repertoire architecture may emerge, where highly connected clones are self-directed as a natural consequence of ontogenetic learning.

Extensive parallel processing on scale-free networks.

We adapt belief-propagation techniques to study the equilibrium behavior of a bipartite spin glass, with interactions between two sets of N and P=αN spins each having an arbitrary degree, i.e., number of interaction partners in the opposite set. An equivalent view is then of a system of N neurons storing P diluted patterns via Hebbian learning, in the high storage regime. Our method allows analysis of parallel pattern processing on a broad class of graphs, including those with pattern asymmetry and heterogeneous dilution; previous replica approaches assumed homogeneity. We show that in a large part of the parameter space of noise, dilution, and storage load, delimited by a critical surface, the network behaves as an extensive parallel processor, retrieving all P patterns in parallel without falling into spurious states due to pattern cross talk, as would be typical of the structural glassiness built into the network. Parallel extensive retrieval is more robust for homogeneous degree distributions, and is not disrupted by asymmetric pattern distributions. For scale-free pattern degree distributions, Hebbian learning induces modularity in the neural network; thus, our Letter gives the first theoretical description for extensive information processing on modular and scale-free networks.

Contingent convertible bonds in financial networks.

We study the role of contingent convertible bonds (CoCos) in a complex network of interconnected banks. By studying the system's phase transitions, we reveal that the structure of the interbank network is of fundamental importance for the effectiveness of CoCos as a financial stability enhancing mechanism. Our results show that, under some network structures, the presence of CoCos can increase (and not reduce) financial fragility, because of the occurring of unneeded triggers and consequential suboptimal conversions that damage CoCos investors. We also demonstrate that, in the presence of a moderate financial shock, lightly interconnected financial networks are more robust than highly interconnected networks. This makes them a potentially optimal choice for both CoCos issuers and buyers.

Modelling time-varying interactions in complex systems: the Score Driven Kinetic Ising Model.

A common issue when analyzing real-world complex systems is that the interactions between their elements often change over time. Here we propose a new modeling approach for time-varying interactions generalising the well-known Kinetic Ising Model, a minimalistic pairwise constant interactions model which has found applications in several scientific disciplines. Keeping arbitrary choices of dynamics to a minimum and seeking information theoretical optimality, the Score-Driven methodology allows to extract from data and interpret the presence of temporal patterns describing time-varying interactions. We identify a parameter whose value at a given time can be directly associated with the local predictability of the dynamics and we introduce a method to dynamically learn its value from the data, without specifying parametrically the system's dynamics. We extend our framework to disentangle different sources (e.g. endogenous vs exogenous) of predictability in real time, and show how our methodology applies to a variety of complex systems such as financial markets, temporal (social) networks, and neuronal populations.

Inverse problems for structured datasets using parallel TAP equations and restricted Boltzmann machines.

We propose an efficient algorithm to solve inverse problems in the presence of binary clustered datasets. We consider the paradigmatic Hopfield model in a teacher student scenario, where this situation is found in the retrieval phase. This problem has been widely analyzed through various methods such as mean-field approaches or the pseudo-likelihood optimization. Our approach is based on the estimation of the posterior using the Thouless-Anderson-Palmer (TAP) equations in a parallel updating scheme. Unlike other methods, it allows to retrieve the original patterns of the teacher dataset and thanks to the parallel update it can be applied to large system sizes. We tackle the same problem using a restricted Boltzmann machine (RBM) and discuss analogies and differences between our algorithm and RBM learning.

Inference of the kinetic Ising model with heterogeneous missing data.

We consider the problem of inferring a causality structure from multiple binary time series by using the kinetic Ising model in datasets where a fraction of observations is missing. Inspired by recent work on mean field methods for the inference of the model with hidden spins, we develop a pseudo-expectation-maximization algorithm that is able to work even in conditions of severe data sparsity. The methodology relies on the Martin-Siggia-Rose path integral method with second-order saddle-point solution to make it possible to approximate the log-likelihood in polynomial time, giving as output an estimate of the couplings matrix and of the missing observations. We also propose a recursive version of the algorithm, where at every iteration some missing values are substituted by their maximum-likelihood estimate, showing that the method can be used together with sparsification schemes such as lasso regularization or decimation. We test the performance of the algorithm on synthetic data and find interesting properties regarding the dependency on heterogeneity of the observation frequency of spins and when some of the hypotheses that are necessary to the saddle-point approximation are violated, such as the small couplings limit and the assumption of statistical independence between couplings.

Detectability of macroscopic structures in directed asymmetric stochastic block model.

We study the problem of identifying macroscopic structures in networks, characterizing the impact of introducing link directions on the detectability phase transition. To this end, building on the stochastic block model, we construct a class of nontrivially detectable directed networks. We find closed-form solutions by using the belief propagation method, showing how the transition line depends on the assortativity and the asymmetry of the network. Finally, we numerically identify the existence of a hard phase for detection close to the transition point.

Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors.

Restricted Boltzmann machines are described by the Gibbs measure of a bipartite spin glass, which in turn can be seen as a generalized Hopfield network. This equivalence allows us to characterize the state of these systems in terms of their retrieval capabilities, both at low and high load, of pure states. We study the paramagnetic-spin glass and the spin glass-retrieval phase transitions, as the pattern (i.e., weight) distribution and spin (i.e., unit) priors vary smoothly from Gaussian real variables to Boolean discrete variables. Our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard Hopfield model with Boolean patterns. The retrieval region becomes larger when the pattern entries and retrieval units get more peaked and, conversely, when the hidden units acquire a broader prior and therefore have a stronger response to high fields. Moreover, at low load retrieval always exists below some critical temperature, for every pattern distribution ranging from the Boolean to the Gaussian case.

Phase transitions in restricted Boltzmann machines with generic priors.

We study generalized restricted Boltzmann machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as generalized Hopfield models. We underline the role of the retrieval phase for both inference and learning processes and we show that retrieval is robust for a large class of weight and unit priors, beyond the standard Hopfield scenario. Furthermore, we show how the paramagnetic phase boundary is directly related to the optimal size of the training set necessary for good generalization in a teacher-student scenario of unsupervised learning.

Emerging Heterogeneities in Italian Customs and Comparison with Nearby Countries.

In this work we apply techniques and modus operandi typical of Statistical Mechanics to a large dataset about key social quantifiers and compare the resulting behaviors of five European nations, namely France, Germany, Italy, Spain and Switzerland. The social quantifiers considered are i. the evolution of the number of autochthonous marriages (i.e., between two natives) within a given territorial district and ii. the evolution of the number of mixed marriages (i.e., between a native and an immigrant) within a given territorial district. Our investigations are twofold. From a theoretical perspective, we develop novel techniques, complementary to classical methods (e.g., historical series and logistic regression), in order to detect possible collective features underlying the empirical behaviors; from an experimental perspective, we evidence a clear outline for the evolution of the social quantifiers considered. The comparison between experimental results and theoretical predictions is excellent and allows speculating that France, Italy and Spain display a certain degree of internal heterogeneity, that is not found in Germany and Switzerland; such heterogeneity, quite mild in France and in Spain, is not negligible in Italy and highlights quantitative differences in the habits of Northern and Southern regions. These findings may suggest the persistence of two culturally distinct communities, long-term lasting heritages of different and well-established customs. Also, we find qualitative differences between the evolution of autochthonous and of mixed marriages: for the former imitation in decisional mechanisms seems to play a key role (and this results in a square root relation between the number of autochthonous marriages versus the percentage of possible couples inside that country), while for the latter the emerging behavior can be recovered (in most cases) with elementary models with no interactions, suggesting weak imitation patterns between natives and migrants (and this translates in a linear growth for the number of mixed marriages versus the percentage of possible mixed couples in the country). However, the case of mixed marriages displays a more complex phenomenology, where further details (e.g., the provenance and the status of migrants, linguistic barriers, etc.) should also be accounted for.

Topological properties of hierarchical networks.

Hierarchical networks are attracting a renewal interest for modeling the organization of a number of biological systems and for tackling the complexity of statistical mechanical models beyond mean-field limitations. Here we consider the Dyson hierarchical construction for ferromagnets, neural networks, and spin glasses, recently analyzed from a statistical-mechanics perspective, and we focus on the topological properties of the underlying structures. In particular, we find that such structures are weighted graphs that exhibit a high degree of clustering and of modularity, with a small spectral gap; the robustness of such features with respect to the presence of thermal noise is also studied. These outcomes are then discussed and related to the statistical-mechanics scenario in full consistency. Last, we look at these weighted graphs as Markov chains and we show that in the limit of infinite size, the emergence of ergodicity breakdown for the stochastic process mirrors the emergence of metastabilities in the corresponding statistical mechanical analysis.

Hierarchical neural networks perform both serial and parallel processing.

In this work we study a Hebbian neural network, where neurons are arranged according to a hierarchical architecture such that their couplings scale with their reciprocal distance. As a full statistical mechanics solution is not yet available, after a streamlined introduction to the state of the art via that route, the problem is consistently approached through signal-to-noise technique and extensive numerical simulations. Focusing on the low-storage regime, where the amount of stored patterns grows at most logarithmical with the system size, we prove that these non-mean-field Hopfield-like networks display a richer phase diagram than their classical counterparts. In particular, these networks are able to perform serial processing (i.e. retrieve one pattern at a time through a complete rearrangement of the whole ensemble of neurons) as well as parallel processing (i.e. retrieve several patterns simultaneously, delegating the management of different patterns to diverse communities that build network). The tune between the two regimes is given by the rate of the coupling decay and by the level of noise affecting the system. The price to pay for those remarkable capabilities lies in a network's capacity smaller than the mean field counterpart, thus yielding a new budget principle: the wider the multitasking capabilities, the lower the network load and vice versa. This may have important implications in our understanding of biological complexity.