{'fraction_non_alphanumeric': 0.039346870029007205, 'fraction_numerical': 0.013755029475063161, 'mean_word_length': 4.794253185145026, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Thermodynamics and evolutionary biology through optimal control

We consider a particular instance of the lift of controlled systems recently proposed in the theory of irreversible thermodynamics and show that it leads to a variational principle for an optimal control in the sense of Pontryagin. Then we focus on two important applications: in thermodynamics and in evolutionary biology. In the thermodynamic context, we show that this principle provides a dynamical implementation of the Second Law, which stabilizes the equilibrium manifold of a system. In the evolutionary context, we discuss several interesting features: it provides a robust scheme for the coevolution of the population and its fitness landscape; it has a clear informational interpretation; it recovers Price equation naturally; and finally, it extends standard evolutionary dynamics to include phenomena such as the emergence of cooperation and the coexistence of qualitatively different phases of evolution, which we speculate can be associated with Darwinism and punctuated equilibria.


I. INTRODUCTION
In Thermodynamics, the description and the control of complex open systems are fundamental problems with direct applications in engineering and in chemistry. Recently, the dynamical behavior of such systems has been formulated using contact Hamiltonian dynamics, that is, by means of dynamical systems generated by exploiting the geometric properties of equilibrium thermodynamics [1][2][3][4][5]. This formulation is particularly suited since it permits to explicitly take into account all the relevant constraints and express the mathematical properties in terms of physically relevant quantities. In particular, in [3] the concepts of "conservative contact systems", and "conservative control contact systems" have been introduced. Moreover, in [6][7][8][9][10][11][12] the properties of controllability and stability of the flow have been thoroughly analyzed.
In Evolutionary Dynamics, it is generally recognized that the main building blocks of evolution are replication, selection and mutation [13,14]. Replication and natural selection can be mathematically formulated using the so-called Replicator Equation, which basically formalizes the intuitive concept that those individuals (or strategies) which have a larger fitness prevail over those that have a smaller fitness and thus eventually dominate the population. In order to include mutations, one can consider the quasispecies model or the more general replicator-mutator equation [13,14].
A major drawback in the formulation of evolutionary biology in terms of replicator-type dynamics with a fixed fitness landscape is the lack of predictive power. As it has been argued previously, fitness is only an a posteriori information and even a "Darwinian demon" who knows the fitness landscape perfectly at present time, would not be able to make reliable predictions on the future of the population, due to the endogenous production of new species by mutations or to the response to changes in the external conditions. To overcome such problem, a coevolution of the existing species and their fitness landscape is desirable [15][16][17][18]. Another subject of major interest in evolutionary biology is the fact that evolution is similar to an optimization process: a population evolves in time in order to adapt as much as possible to some external conditions [19][20][21]. A final relevant aspect in evolution is understanding the role played by information, that is, to formulate rigorously the process of adaptation as a learning mechanism. In this sense a remarkable result was proved by Frank, who showed that natural selection maximizes Fisher information [22,23] (see also [24][25][26][27][28][29][30][31][32][33] for further discussions relating information theory, statistical physics, thermodynamics and evolutionary biology).
The aim of this work is to regard evolution, either of a thermodynamic system or of a population of individuals, as an adaptive process in which the system dynamically controls some parameters in order to evolve towards the optimum state allowed by the constraints. To do so, first we review and generalize a specific instance for the lift of irreversible thermodynamic systems proposed in [3,[6][7][8][9][10][11][12] and prove that such lift is equivalent to an optimal control in the sense of Pontryagin [34].
Then we show that this optimal control provides both a dynamical implementation of the Second Law and a practical method to stabilize the equilibrium manifold for a generic thermodynamic system. After that, we apply this technique to evolutionary dynamics. In particular, we take the lift of the Replicator Equation and show that this amounts to constructing an optimal control over the fitness parameters that maximizes the accumulated covariance of the trait of interest with respect to the fitness landscape. This provides a variational principle for evolutionary dynamics that solves the problem of the lack of predictive power of the Replicator Equation, as it can be used to evaluate dynamically the change in the fitness due to spontaneous mutations and variations in the external conditions. Moreover, we interpret our principle in terms of the maximization of Fisher's information proposed by Frank [22,23] and we show that Price equation emerges naturally in this context. Finally, we comment on the fact that such optimal control strategies can describe experimentally observed data from protein reactions [20], as well as the emergence of cooperation in social dilemmas [35], and we speculate that in some cases the different dynamical phases stemming from the optimal control could be used to provide a unified dynamical scheme for the different ("Darwinian" and "punctuated equilibria") phases of natural evolution [36][37][38][39].

II. ABOUT THE LIFT OF CONTROLLED SYSTEMS
Let us start by reviewing briefly (and extend in its scope) the lift of controlled dynamical systems proposed recently in [3] (see also [6][7][8][9][10][11][12]). We consider a dynamical system of interest, represented by the equationsẋ where x = (x 1 , . . . , x n ) is the vector of quantities describing the state of the system, and u = where x is the vector of extensive variables of the system, S(x) is the fundamental relation for the entropy, and ∇ x S is the vector of intensive quantities. Here d is called the drift vector and (c 1 , . . . , c m ) are the control vectors, representing the interaction between the system and the environment. We note also that the system (2) is a very special case of (1), since it is affine in the vector of control parameters u. Systems of this type are referred to as systems affine in the control.
From (2) we see that in many cases the dynamics of interest is constrained to a surface which can be described by the graph of a function, that is, there exists a function ψ(x) such that (1) can be written asẋ In such cases it is convenient to enlarge the state space with coordinates x to an Extended Phase Space (EPS) of dimension 2n + 1, with coordinates (x, p, z), and define an appropriate "Hamiltonian" function h(x, p, z) so that its "Hamilton equations" are such that equations (4) coincide with (3) on the surface given by the graph of ψ, namely on This procedure is called the lift of the system to the EPS 1 . Notice that whenever the Hamiltonian, h, does not depend on z, as it will be the case in this work, the system (4)-(5) reduces to the standard Hamilton equations, together with equation (6) for the "action" [42][43][44][45][46][47].
These lifts are particularly meaningful for thermodynamic systems of the type (2). In such context, one can think of p as the vector of possible values for the intensive parameters of the reservoirs and conclude that the conditions in (7) are the conditions for thermodynamic equilibrium.
A desirable property for the lift of a system to the EPS is that the surface L ψ be invariant, meaning that once the system enters such surface it remains on it. In this way one can be sure that the lifted flow preserves the relations of interest (7). Fortunately, this property can be easily implemented in the EPS. In fact, it can be proved that L ψ is invariant for (4)- (6) if and only if h| L ψ = 0, that is, if and only if h vanishes on L ψ [1,3,10]. According to a definition given in [3], lifted systems that satisfy such property are called conservative with respect to L ψ .
From the above description it turns out that, given a system (1) and a function ψ(x), there exist many possible conservative lifts, and currently there is no principled way to distinguish them [10].
Here we consider a natural candidate h ψ to define a conservative lift: extending the definitions in [3,10] to the more general context of systems as (1), we define From (4) it follows thatẋ = F(x, u) and therefore h ψ defines a lift of the system to the EPS.
Besides, clearly h ψ | L ψ = 0, which means that the lift is conservative. Indeed, since (8) depends on the external (controllable) parameters u, this system is a conservative control contact system [3,7].

A. Pontryagin's Minimum Principle
The typical fixed time optimal control problem is formulated as follows [34]: given U ⊆ R m , F : R n × U → R n and x 0 ∈ R n , define the set of admissible controls as Then, given the controlled systemẋ and the cost functional the basic problem of optimal control theory is to find the optimal control u * such that Here L(x, u) is the analogue of the Lagrangian for a mechanical system and g(x(t f )) is the final cost. Pontryagin's Minimum Principle (PMP) states that the optimal strategy u * is to be found by introducing the co-state variables p ∈ R n and the control Hamiltonian and then solving the corresponding system of Hamilton equationṡ together with the minimum condition for all 0 ≤ t ≤ t f and the terminal condition B. Lifted systems as optimal protocols Provided (1) and (10) (8) and H in (13) and infer that, if u(t) is chosen so that h ψ is minimized at every t as in (15), then the lifted system (4)-(5) is equivalent to Pontryagin's equations (14) for the cost Besides, by (8), for z(0) = ψ(x(0)) equation (6) is equivalent to Therefore we arrive at the following proposition: the equations (4)-(6) are Pontryagin's equations for the optimal states and co-states of the problem subject to the constraint (1) 2 . The minimum and the terminal conditions are given by (15) and (16) respectively. We stress that this is a special feature of the particular lift (8) that we have chosen, and thus this property may help to single out this lift among the many possible equivalent lifts of a given system (cf. [10]).
In the particular case of the balance equations (2), the potential ψ(x) = S(x) being the entropy, choosing g(x(t f )) = S(x(t f )), equation (19) reduces to Since the initial state of the system (and thus its initial entropy) is fixed by the initial condition (cf. (10)), the above principle amounts to maximizing the final entropy of the system subject to the dynamical constraint (1).
We remark that by (16) we have p(t f ) = ∇ x S(x(t f )) and by (18) it follows that z(t f ) = S(x(t f )), that is, the resulting final state is always an equilibrium state (cf. (7)). In other words, we get a dynamics consistent with the Entropy Maximum Principle, which is one of the standard formulations of the Second Law of thermodynamics [53].
Moreover, as a by-product, we gain two interesting results: we have managed to stabilize the equilibrium manifold, which is by itself a very important problem in the theory of control of thermodynamic systems that has attracted much attention in recent years (cf. the discussion in [3, 6-12]); we have interpreted the equilibrium conditions in thermodynamics as the terminal conditions (16) in PMP.
To conclude, we note that in some important cases the balance equations are affine in the control parameters as in (2), and U is of the form In such cases the optimal protocol for u is a bang-bang control [34] 3 , for which the switch condition is possibly augmented with a singular control for

A. Evolution by fitness control
Let us now apply the previous ideas to the Replicator Equation (RE) [13,14] x Here each x a is the proportion of the population of type a (in evolutionary game theory it is the proportion of the population adopting the strategy a), each f a (x) is the (frequency-dependent) fitness of type a, meaning its ability to generate offspring, and f = n i=1 x i f i is the average fitness of the population. Moreover, in the last equality in (24) we have introduced a notation which is useful in order to compare with the above description of control theory 4 . In fact, usually in evolutionary dynamics the fitness vector f = (f 1 , . . . , f n ) is assumed to be a pre-assigned function of x. However, this leads to the problem of the lack of predictive power of equation (24). Here instead, we treat f as a control parameter and apply the conservative lift presented above. In this way we obtain a variational principle for the RE, together with an optimal protocol for the evolution of the fitness.

B. The Optimal Replicator Equation
Let us start by considering a trait of interest τ , together with its average value Using ψ τ given by (25) and the corresponding Hamiltonian h ψτ as in (8), we can construct a conservative lift.
From the analysis of the previous section, provided the components of f are chosen at any time so that h ψτ is minimized, the system (4)-(5) thus obtained defines the optimal trajectories for the states and co-states of the problem We observe that (for τ not depending on x) ∇ x i ψ τ = τ i , and that by (24) and using conservation of probability, we can write The key point here is that the term on the right hand side of equation (27) is the covariance between the trait τ and the fitness f . We conclude that the conservative lift of the RE associated with the trait τ introduced here defines Pontryagin's equations for the following optimization problem We propose this principle as a generalization of the standard RE, and we call the system of equations (14)-(16) resulting from the associated PMP the Optimal Replicator Equation (ORE).
The principle (28) can be interpreted as follows: the system during its evolution updates both its fitness values and the proportion of the population adopting each strategy according to the RE and with the objective of maximizing the accumulated covariance of a trait with respect to the fitness.
Let us discuss some ramifications of this principle that can be confronted with previous theoretical and experimental work: since the covariance cov(τ , f ) can also be interpreted in terms of Fisher information, our result (28) can also be explained as the fact that the system evolves in order to maximize its accumulated Fisher information, similarly to what was already argued in [22,23]. Furthermore, the system (4)-(6) has to be augmented with the minimum condition (15) for h ψτ and the terminal condition (16) in order to obtain the full set of requirements in Pontryagin's theorem. Our goal is to argue that the resulting controlled dynamics provides an extension of the standard RE that can help describe some observed natural phenomena. To do so, we consider two cases: first, if τ = −f , the Hamiltonian h ψτ is quadratic in the controls and the optimal control can be found analytically. In such a case, by choosing g(x(t f )) = − f (t f ) , we directly recover the form of the ORE recently introduced in [35] in order to explain the emergence of cooperative behavior among selfish individuals in a way similar to the model discussed in [54,55]. Secondly, whenever ψ τ does not depend on f , the Hamiltonian h ψτ is affine in the controls. Moreover, since the fitnesses can be safely assumed to be bounded, we have that the optimal protocol is given by a bang-bang control on the fitness values, similar to (22). Interestingly, a similar bang-bang behavior has been reported in [20], where it has been shown that it matches data from protein reactions, providing further evidence that these types of strategies can indeed describe the adaptive evolution of living systems.
Another interesting aspect is that such controls, together with equations (4)-(6), provide a consistent scheme for the coevolution of the system and its fitness values over time, which is robust with respect to spontaneous changes in the fitness landscape due to mutations or production of new species. Therefore such mechanisms can also be used to evaluate the response of the population to mutations or changes in the environmental conditions. We will address this case in a future work by considering the replicator-mutator equation.
Finally, in the case of the bang-bang control, we can imagine qualitatively the dynamics as follows: whenever the fitness is constant, there is a standard evolutionary phase, governed by the RE with fixed values of f . Eventually a change occurs which drives the dynamics away from this equilibrium phase. This means that the corresponding parameters f a start to follow the bang-bang protocol, with possible jumps between the maximum and minimum allowed values. It is tempting to speculate that the first phase can be associated with Darwinism, meaning that changes are smooth and dictated only by natural selection, while the second phase can be associated with punctuated equilibria, meaning that the fitness of each species (and the very presence of the species) can change abruptly, with the dynamics not completely determined by the RE [36][37][38][39]. In this sense our optimization principle may provide a unified dynamical scheme for these two complementary theories of evolution. Future work will be pursued in this direction.

C. Price equation as a conservation law
In order to further support our optimization scheme for evolutionary dynamics encoded in the variational principle (28), let us remind that the dynamics resulting from Pontryagin's theorem can be alternatively seen as the conservative lift of the RE to the EPS. In this perspective, let us consider in detail the condition for the lift to be conservative with respect to the surface defined by the trait ψ τ according to (7). First notice that Then we compute the time derivative of ψ τ Thus, from (29) and (30), we can rewrite the condition for the lift to be conservative with respect to the surface generated by ψ τ -that is, h ψτ | L ψτ = 0 -as which is exactly Price equation for the evolution of the trait τ [13,56]. Notice that whenever the fitness f is constant, i.e.ḟ = 0, Fisher's fundamental theorem of natural selection d dt f = var(f ), follows from (31) by taking the particular case in which the trait of interest is just τ = ±f .

VI. CONCLUSIONS AND FURTHER DIRECTIONS
The formalism presented here, based on the lift of controlled systems and on the variational principle arising from comparison with Pontryagin's theorem for optimal control, can be applied to a great variety of systems and situations. In particular, we have considered two important cases: the irreversible evolution of thermodynamic systems, and the description of population dynamics in terms of optimal control.
For thermodynamic systems, we have shown, as a first main result, that our principle describes a dynamical implementation of the Second Law, which in turns provides a direct stabilization of the equilibrium manifold.
Regarding evolutionary dynamics, we have obtained, as a second main result, an extension of the Replicator Equation in terms of our variational principle (28). According to such principle natural evolution can be phrased as an adaptation process in which the system maximizes the accumulated information (codified in the covariance of a specific trait with respect to the fitness landscape), by dynamically adjusting the values of the fitness parameters at any time. We highlighted several consequences of this principle which are of special interest for evolutionary biology: it resolves the problem of the lack of predictive power of the Replicator Equation; it provides an analogue of the informational principle of maximization of Fisher information put forward by Frank [22,23]; the resulting optimal strategies can help describe the emergence of cooperation [35] as well as of experimentally observed phenomena occurring in protein reactions [20]; finally, Price equation is recovered naturally as the condition for the system to be conservative.
Our scheme can be extended in several directions. On the one hand, we believe that the construction of the Second Law as an optimization process may have direct applications in the stabilization of open thermodynamic systems [10]. On the other hand, although we have considered here only deterministic equations, optimal control theory may be applied also to stochastic equations, and thus our proposal can be compared with similar and more realistic models including the stochastic drive (see [25,26] and references therein). Moreover, our work shows an intriguing connection among thermodynamics, evolutionary biology, and information theory that has been only scratched here. Finally, as explained at the end of Section V, we consider that our framework will be especially useful to re-formulate in a unified scheme the theories of punctuated equilibria and that of natural selection.