{'fraction_non_alphanumeric': 0.0295551492992078, 'fraction_numerical': 0.005223295899712719, 'mean_word_length': 4.638036809815951, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
From inverse problems to learning: a Statistical Mechanics approach

We present a brief introduction to the statistical mechanics approaches for the study of inverse problems in data science. We then provide concrete new results on inferring couplings from sampled configurations in systems characterized by an extensive number of stable attractors in the low temperature regime. We also show how these result are connected to the problem of learning with realistic weak signals in computational neuroscience. Our techniques and algorithms rely on advanced mean-field methods developed in the context of disordered systems.


Introduction
The enormous amount of data which is rapidly accumulating across scientific disciplines has given to inverse problems a renewed fundamental role. For instance, the problem of inferring effective or direct interactions in systems composed of many heterogeneous degrees of freedom is at the root of many fields, ranging from natural sciences to social sciences. Quite often, the systematic experimental measurements of the interactions are difficult to perform, and to estimate them one needs to resort to indirect statistical inference methods. These need to be computationally efficient in order to process experimental datasets which are typically huge. During the last decade, statistical inference challenges have stimulated lots of research efforts in the statistical physics community [1] in the attempt to adapt rigorous theories and algorithms to the problem of going from observables to parameters, somehow the inverse of what statistical physics has mostly been developed for. Spectacular examples of inverse problems can be found in the biological sciences where experimental data on neuronal populations [2,3], proteins [4], biomolecules and genetic interactions [5], have largely surpassed the understanding of the underlying complex chemical mechanisms. In physics too there exist many examples of inverse statistical problems, e.g. the design of many-body systems with particular desired properties (e.g. [6], see refs. [1]) or to learn from data some effective parametrizations of wave-functions (e.g. [7]). The technological progress is rapidly opening up many more applications.
Inverse problems and statistical inference are closely linked to machine learning. Here we will focus on unsupervised learning, i.e. extracting information from unlabeled data [8]. The vast majority of the data are indeed unlabeled, and unsupervised learning is possibly the most challenging branch of machine learning, which has a direct mapping onto maximum entropy ideas [9] and statistical mechanics methods. Adopting the physics notation, the unsupervised learning problem can be stated as follows. We assume that a set of M i.i.d. sampled configurations D = σ (μ) μ=1,...,M drawn from an unknown distribution P (σ) = 1 Z e −H(σ) are given. The configurations components σ (μ) i can be either continuous or discrete. We want to identify a probability distribution over the energy functions H (σ) which is compatible with the data. The likelihood of the model given the data is defined as the posterior distribution P ( is the average over the data and P (H) is the prior about H. The number of samples M plays a crucial role, quite similar to inverse temperature: when M is large, P (H|S) concentrates on the maximums with respect to H of the log-likelihood L = H + log Z. Inferring the most probable H from the data amounts at identifying the maximum of L. This is a formidable optimization problem, due to the difficulty of estimating Z on a candidate solution for H.
In what follows, for the sake of simplicity, we assume that the dynamical variables {σ} are binary, i.e. Ising spins. The simplest non-trivial scenario assumes that we can have access to random configurations which allow us to measure non vanishing pairwise correlations between spins. In absence of correlations, the problem trivializes. In a general setting, the couplings characterizing the H (σ) we want to infer will not all be positive. Moreover they will be embedded in some high-dimensional space. A concrete working example is provided by a system of N spins {σ i ≡ s i = ±1} which interact according to a generic (unknown) spin glass like Hamiltonian, i.e. spins are coupled by pairwise couplings J ij and are subject to external magnetic fields h i . The unknown distribution from which the random configurations (the data) are extracted is a Boltzmann distribution where we have included the inverse temperature factor into the couplings and fields. The problem we want to solve is the determination of the most probable values for the couplings J ij and local fields h i , given a set of M observed spin configurations. Depending on the particular nature of the system at hand, the functional form of the Hamiltonian may be need to include higher order terms. Or even the choice of the form of the probability distribution may not be appropriate (e.g. if the systems are out of equilibrium). Still, the basic ideas are already captured by the above Ising like choice. It is in fact well known that the distribution (1) is not only the equilibrium distribution of the Ising model but, more importantly in our context, it is also the form of the distribution which maximizes the (Gibbs) entropy with P ({s i }) satisfying the constraints of normalization and of having first and second moments (magnetizations and correlations) that match some given experimentally measured values. The maximum entropy P ({s i }) is thus the probability distribution which matches the constraints given by some observables, without making any further assumption on the data, i.e. being the least biased distribution [10]. In settings where higher moments are not captured by (1) one must resort to maximum entropy distributions which match such moments and have multi-body interactions. Finding the parameters of (1) is a delicate computational problem: any small change in the couplings affects correlations between many spin variables and vice versa. Determining correlations for a given value of the parameters is a difficult computational problems which needs to be repeated multiple times in the search for the optimal couplings. Several techniques can be In what follows we will provide some preliminary results in which we discuss how the above ideas can be used to infer couplings in systems characterized by multiple attractors. As a byproduct we will show how the overall technique can lead to some interesting learning mechanisms which might be of interest in modeling neural systems. We will perform sampling by a mean-field technique developed in the context of disordered systems [11]. Still other approaches, e.g. Monte Carlo Markov chains, could be used.

Attractor networks
The celebrated Hopfield model [12] suggests a way of modeling associative memory as a process through which a set of M memories {ξ μ } M μ=1 are stored within a recurrent neural network as the minima of an Ising-like Hamiltonian. The network is modeled as consisting of N binary stochastic units representing the neurons. At each time t, a unit state s t i ∈ {−1, 1} (representing whether the neuron is active or quiescent) is determined from the other units according to the probability distribution where β controls the level of stochasticity, the quantities J ij model the synaptic connection strengths and h ext i some external input. As β → ∞, the neurons become deterministic and their state is simply given by the sign of the total local field h tot acting upon them. The above equation implements a parallel dynamics, in which time is discretized and all units are updated simultaneously; it is straightforward to implement other update protocols (e.g. random sequential); in such case, and if the synaptic connections are assumed to be symmetric, J ij = J ji , the probability distribution of the whole network matches Eq. (1) with h = h ext . This insight allows one to use the tools from statistical mechanics to study the behavior or the network for given J and h ext .
In its original formulation, the Hopfield model assumed that the synaptic couplings are shaped according to the Hebb's learning rule, reinforcing the connections inducing positive correlations among neurons and depressing the couplings leading to negative ones. Then, if the network is presented, during a training phase, with a number M of binary input patterns ξ μ (by setting in succession h ext i = λξ μ i for all μ, with λ 1 such that the external fields completely clamp the state of the network in Eq. (3) to s t = ξ μ ) the resulting couplings will have the symmetric form It can then be shown that, for sufficiently high β and sufficiently low M , there exist a regime in which this model behaves as an attractor network, meaning that initializing the state of the network in the proximity of any of the patterns, i.e. s 0 = ξ μ + ν with ν some binary noise that inverts some fraction of the units, the dynamics of the network, Eq. (3), spontaneously converges towards the basin of attraction of the memory to be stored, s t ∼ ξ μ for t 1. The amount of noise ν for which this is possible determines the size of the basin of attraction of the patterns. In this way, the input patterns {ξ μ } are said to be stored as (associative) memories in the network, since they are encoded in the J matrix and can be recovered from partial or corrupted inputs.
The Hopfield model with the Hebb's learning rule has notoriously a number of problematic features [13], among which the most prominent are: a rather low capacity (the maximum M scales as 0.14N , while it is known that a network could reach 2N if β → ∞ and if the symmetry constraint on J is dropped); the existence of attractors of the dynamics that were not in {ξ μ } 4 1234567890 ''"" (called "spurious" attractors); the phenomenon of catastrophic forgetting (when M exceeds 0.14N the network enters a spin-glass phase in which all the ξ μ attractors are suddenly forgotten and an exponential number of spurious attractors are present instead); the inability to deal in general with correlated input patterns (although Hebb's rule can be amended in some simpler cases, e.g. for biased patterns); the requirement of very strong external fields during the training phase (expressed as λ 1 above), which is somewhat biologically implausible, while the learning rule requires that each unit should somehow be aware of both its current state (clamped to ξ μ i ) and the state that it would have in absence of its own external field, which is even more biologically problematic.
The choice of using Hebb's rule was motivated by its simplicity and by biological considerations; moving away from this choice allows to amend many of the above-mentioned issues (see e.g. Refs. [14,15]). Here, we extend the learning protocol proposed in Ref. [16], which specifically addressed the problem of learning with small external fields, to fully-connected networks.
As shown in more detail in the next section, the approach frames the learning problem as an inverse problem, in which the memories {ξ μ } represent the given data D, and we seek to find a coupling matrix J such that the memories would be typical samples from the distribution of Eq. (1), i.e. representatives of the basins of attraction of the final model.

Description of the learning protocol
Generalizing the work of Ref. [16], we study the problem of learning stable attractors by employing finite signals, in a fully-connected recurrent neural network. The inverse problem, i.e. the unsupervised learning task, is defined as follows: each pattern ξ μ , μ = 1, ..., αN , is presented to the network in the form of an external field of intensity λ, oriented along the direction of the pattern itself, namely h ext = λξ μ . The relative balance between the intensities of the recurrent and the external contributions to the local field h i = j =i J ij s j + h ext , can produce a dynamics which goes from a clamped regime, where the state of the network is set by the external field in a deterministic fashion, to a noise dominated regime, where the dynamics of the network becomes fully stochastic. The goal of the learning rule is that of adapting the coupling matrix J (i.e., the synaptic couplings), so that the driving effect of the external field is effectively absorbed in the recurrent interaction of the network. In this way, the freely evolving dynamics of the network is required to maximally resemble the driven dynamics.
This prescription naturally translates into a simple learning rule, inspired by the close connection between learning and inference in inverse problems: where the parameter η plays the role of a learning rate. The analogy with the Inverse Ising problem is immediately evident: the difference between the empirical correlations and the ones estimated from the partition function of the inferred model, is here replaced by the correlation difference between two models with the same synaptic couplings J, but subject to different external field intensities, λ 1 and λ 2 . However, we should note that, contrary to what happens in inference problems, the two correlations will never perfectly match.
As in Ref. [16], we note that in the limit of high temperatures and very strong biasing signals, i.e. λ 1 → ∞, we can recover the classic Hebb rule, since s i s j 0 0 and ΔJ ij ∝ s i s j ∞ = ξ μ i ξ μ j . At lower temperatures, in the same clamping limit, the learning protocol translates into the so-called Contrastive Divergence algorithm Ref. [17].
The computational problem associated with this learning strategy is that of the correlations estimation: in the following sections we propose a mean-field method developed in the Spin Glass theory framework. Randomly initialize ξ μ , μ = 1, ..., p with ξ μ i = {−1, 1}; Initialize J s to zero; Choose a field intensity λ 1 ; Set a field intensity λ init for μ in random permutation of p patterns do Initialize magnetization close to the choosen pattern μ; Algorithm 1: Learning

Correlations recording
In the previous section we have seen how the analogy between inference and learning leads to a new learning protocol, according to which the parameters of the network, namely the synaptic couplings J ij , have to be modulated in order to allow the storage of a set of p memories. The resulting update rule relies on the matching between the pair-wise correlations, estimated in presence of an external field of intensity λ 1 , and the ones recorded in presence of a weaker external field λ 2 , progressively vanishing. The central issue is then how to compute the pair-wise correlations. In some contexts, more biologically inspired, the evolution in time of the network is simulated through the Glauber dynamics. The correlations among neuronal state variables are then estimated by means of the Monte Carlo method. However, it is known that this procedure becomes unfeasible for too large network sizes: the time needed for reaching a stationary state consistently increases when a bigger number of network components comes into play. We thus need to resort to more efficient methods.
In this work, we take advantage of the Belief Propagation algorithm (BP). This meanfield cavity method provides an estimation of the marginals associated to the variables that characterize a given physical system. Despite the fact that this estimation is exact only in treelike graphical models, BP can be still successfully employed on loopy-graphs, when they can be locally approximated as trees.
By computing the (cavity) marginals, other interesting physical observables, such as the pairwise correlations, can then be estimated. The BP algorithm accomplishes this task by defining a set of messages (also called cavity beliefs) running across the edges of the graph and corresponding to the marginals of a given variable in absence of a specific node of the graph (i.e. in presence of a "cavity"). The marginals can then be obtained as the product of the messages over all the cavity sites. BP provides a set of self-consistent equations that need to be solved iteratively, in order to find a closed expression for the cavity beliefs.
In the model of a fully-connected neural network, the BP equations of the messages are known, and given by: The cavity message π i→j (s i ) then determines the marginal of the s i variable in absence of the node j in the network. In the thermodynamic limit, we can get the asymptotic expression: where we have defined m i→j as the cavity magnetization: This self-consistent system of equations can be solved by iteration. Once a fixed point is reached, the pair-wise connected correlations can be recovered, on the basis of the cavity magnetization, as:

The TAP Approach
The BP algorithm is a very powerful tool. As we have seen, once the cavity messages are known, we are able to derive the main interesting observables characterizing a physical system, such as the pair-wise correlations. However, one of the drawbacks of this method is that BP requires the storage of a number O N 2 of cavity messages, thus considerably affecting the computational performances. In order to overcome this difficulty, we can resort to the Thouless-Anderson-Palmer (TAP) approach. It can be derived through the high temperature expansion, or, equivalently, the limit of small synaptic couplings, of the mean field equations, on which the BP algorithm is based.
By performing the expansion, we get the TAP self-consistent equations, to be solved iteratively: where the time indices have been assigned so as to ensure the convergence of the iterative scheme [18]. We can notice that, instead of involving the cavity magnetizations, the TAP equations refer directly to the magnetizations, thus reducing the number of equations to be solved to O (N ).
In this framework, the pair-wise correlations can be computed as: When running the simulations at a fixed number of learning cycles T L , we can observe that learning with weaker fields is not only possible, but allows the network to reach higher storage loads. The points are obtained by averaging over 20 samples.

Asymmetric Synaptic Couplings
The Hebb's learning rule is characterized by the unrealistic feature of symmetric synaptic couplings, namely J ij = J ji : there is no reason to assume that the signal sent by neuron i towards neuron j has to be of the same strength of the one sent by neuron j towards neuron i. In fact, kinetically persistence states can be still observed in asymmetric networks.
In this section we thus consider a recurrent and fully connected neural network characterized by asymmetric synaptic couplings. The main difficulty in dealing with such a kinetic model is represented by the impossibility of defining a Hamiltonian. Therefore, in principle, we are not allowed to naively resort to the mean field approach. Here we follow the solution proposed in Ref. [19], where the authors show how the TAP equations can still be derived from a Kullback-Leibler divergence minimization even in the context of asymmetric synaptic couplings. We extend their results to the case of a parallel neuronal dynamics. The fixed point equations for the marginal probabilities of the neuronal states read: The TAP approach can be thus exploited in the asymmetric context for the computation of the pair-wise correlations, as in [19]. However, in this specific case, we need to slightly modify the update rule of Eq. (4), in favor of the time-delayed correlations s t+1 i s t j : Indeed, by definition, the equal time correlations considered so far are not the proper quantity for learning in the asymmetric scenario.
Here, we present the results we obtained by applying the new learning protocol to a recurrent neural network, both in the symmetric and in the asymmetric cases. In the retrieval phase, we say that a pattern ξ μ has been stored if the overlap of the stationary state, reached at convergence of the TAP equations, is consistently higher than all the other overlaps with all the other patterns ξ ν , with ν = μ.
In order to guarantee convergence of the TAP equations, instead of directly comparing the λ 1 intensity with λ 2 = 0, we slowly decrease the magnitude of the second external field, starting from an initial value of λ 1 /2, as detailed in Algorithm 1. Fig. 1 shows the number of patterns that have been retrieved by the network, as a function of the intensity of the reference external field λ 1 . Notice that we work at storage loads that are higher then the critical capacity estimated for the Hebb's rule. We can notice that, the network is able to learn even in the presence of weak external field.

Learning correlated patterns
As pointed out above, one of the major drawbacks of the classic Hebb rule is in the inability of embedding correlated memories. We test the learning protocol of Eq. (4) in this setting, introducing an inter-pattern correlation inspired to the hierarchical scheme as discussed e.g. in Ref [20]. In this scheme, the patterns are organized in L categories, each defined by an independently generated random prototype. Each category contains αN/L patterns, obtained as independent perturbations of the corresponding prototype, with a fraction 1 − x of randomly flipped components. By increasing x, one can thus enhance the correlations in the dataset. The purple curve in Fig. 2 shows the maximum storage load α c at which perfect memorization can be achieved, in the case of N = 201 and L = 5, as a function of the correlation x. The reference field intensity λ 1 was set to 0.3. As the learning problem becomes harder, the performance of the networks remains stable before suffering from a sudden drop, for values of x > 0.5.
In order to overcome this problem, we slightly modified the learning prescription, by updating the synapses as: The main intuition behind this adjustment is the following: we are not only requiring the free dynamics of the network to resemble the biased dynamics, but we are also inducing some robustness into the memorization process. In fact, in this case, the network tries to match the initial correlations with the one measured in the presence of an antagonist external field, whose effect needs to be overcompensated by the recurrent couplings of the network. The green curve in Fig. 2 shows the results with the new update rule, in the same settings as above: by implementing this protocol, the network is indeed able to learn even in presence of stronger correlations. neurons. The purple curve shows the learning behavior when the weaker external field λ 2 tends to zero: after the pattern correlation exceeds a given threshold (x = 0.5), the ability of the network to store stable attractors sharply deteriorates. The green curve, instead, shows the case where λ 2 tends to a weak negative value (−0.3): the performance in the highly correlated regime is clearly improved. All the points are obtained by averaging over 20 samples.