{'fraction_non_alphanumeric': 0.026522737263713184, 'fraction_numerical': 0.006398671420895814, 'mean_word_length': 4.762454264002252, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-Organization Promotes the Evolution of Cooperation with Cultural Propagation

In this paper three computational models for the study of the evolution of cooperation under cultural propagation are studied: Kin Selection, Direct Reciprocity and Indirect Reciprocity. Two analyzes are reported, one comparing their behavior between them and a second one identifying the impact that different parameters have in the model dynamics. The results of these analyzes illustrate how game transitions may occur depending of some parameters within the models and also explain how agents adapt to these transitions by individually choosing their attachment to a cooperative attitude. These parameters regulate how cooperation can self-organize under different circumstances. The emergence of the evolution of cooperation as a result of the agent's adapting processes is also discussed.


Introduction
Urban traffic problems have a complex behavior even in their most simplistic abstractions [2]. This behavior becomes more complex when driver interaction is added [10]. With these two premises, in this paper, three computational models, which were originally intended for studying driving behaviors and their impact in traffic, are presented. The aim is to explore the possibilities for controlling some of the complex characteristics of traffic, e.g. the consequences of driver interactions. The possibility of using them for other social problems is also discussed.
The models were conceived to study the evolution of cooperation [1], which studies how cooperative behaviors are preserved when selfish behaviors offer greater individual rewards, e.g. drivers who yield at lane changes or crossings, animal females taking care of their offspring, students who don't betray each other. Several abstractions for simulating different circumstances in which this phenomenon occurs have been proposed [6]. The models are based in three abstractions of different observed situations in which cooperation evolves [4]. Table 1 shows these abstractions as theoretical games in which players may choose between two behaviors: cooperative or selfish. When a player chooses to cooperate she has to pay a cost that her game partner is going to receive as a benefit. In each game, the decision of the players is influenced by a different probabilistic variable. This variable represents some feature that is exploited by the global behavior to favor the evolution of cooperation. For the kin selection strategy, relationships are exploited, the relation may be genetic, emotional or of any other type but as long as it its closer, cooperation will be more frequent. In the direct reciprocity case, the evolution of cooperation is linked with the probability of one player to play again with the same partner, and this condition is set because players are responding cooperation with cooperation and defection with defection. The feature exploited by the indirect reciprocity case is peer pressure. While more players get to know the actions of a determined player, she will obtain more benefits by cooperating and while fewer players get to know her actions more benefits she will obtain by defecting. Each strategy has a condition for the cooperation to become an evolutionary stable strategy (ESS, cooperators survive), another when cooperation becomes risk-dominant (RD, cooperators are near 1/2 of the population) and other when cooperators are advantageous (AD, cooperators are near 2/3 of the population).

Strategy
Payoff  Table 1. Rules for the three game strategies [4].

The Models
We developed agent-based computational models, using the NetLogo [12] platform, to better understand three strategies for the evolution of cooperation: kin selection, direct reciprocity and indirect reciprocity [4]. Unlike the results shown in [4] these models are focused in cultural evolution (players may choose to cooperate or not under different circumstances instead of been born as players who always cooperate or always defect), therefore, the first characteristic these models share is that they have a constant population. The behavior of the agents is similar to the one of agents described in Skyrm's Matching Pennies model [8]. Agents have a cooperation probability variable (cp) that determines their attachment to a cooperator strategy, while this variable gets a greater value the agent will cooperate more frequently. We will use this variable for deciding if the agents will cooperate or not in a particular game. Each agent is able to identify herself as a cooperator or as a defector: Where p i is the i th agent or player, c i is the i th cooperator and d i is the i th defector. As a consequence of this characteristic, a partition of the population can be made. The initial proportions of cooperative versus defective agents in the population (ipc and ipd) are relevant parameters of the model although they can be reduced to one as ipc=1-ipd. In order to identify agents as cooperators or defectors at the beginning of the simulations we give them an initial cp, thus, initial cp of defectors (iicpd) and initial cp of cooperators (icpc) are two more parameters. The agents move through a defined two-dimensional space. It has been seen [3] that in such cases population density is a key determinant of the dynamics, so the number of players (population) interacting in the defined space is another important parameter of the model. Agents interact following the payoff tables corresponding to each strategy studied. As can be seen in Table 1, the three strategies share two variables: the benefit (b) one agent gives to another while cooperating and the cost (c) paid by the cooperator while cooperating. With the purpose of keeping track of the player's decisions, each one of them is assigned a fitness value from where the cost will be subtracted and the benefits will be added.
Appendix A shows the instructions executed by the agents during each iteration of the simulations. The behaviors of each game are detailed in appendix B.

Tune cooperation probability
Agents adapt to the environment by modifying whether they are going to cooperate or defect in the next round. We implemented several exogenous tuning algorithms based on the literature [8] [11] with no satisfactory results. We developed two self-organizing tuning alternatives and were tested with very good results:  Selfish fitness: the agents only take their own fitness value as a parameter, unlike other algorithms that take into account the values of all the other players or values of the game partners. If the player cooperates or defects, and her fitness increases, then the agent increases or decreases her cp respectively. If the player cooperates or defects, and her fitness is reduced, then the agent decreases or increases her cp respectively. And if there is no change in the agent's fitness, then there is no change in her cp.
 Selfish profit: this is very similar to the previous one, but instead of comparing fitness values (sum of all profits obtained), agents act based in the comparison of the profit of their last game and the profit of the current round. Thus, if the player cooperates or defects and the actual profit is greater than the last one, then the agent increases or decreases her cp. If the player cooperates or defects and the actual profit is lower than the last one, then the agent decreases or increases her cp. Finally if the last profit is equal to the actual profit, then there is no change in the cp value.

Methods
Before general results were obtained, an analysis of the variables was conducted. A primary goal of this analysis was to determine the impact of the parameters on the dynamics of the models. A secondary goal was to obtain parameter values that best exemplify the model's behavior. Detailed information about the experiments may be found in appendix C.

Results
Interesting results can be derived from a careful analysis of the graphics of the payoff table behavior as the corresponding probability takes values within the range [0,1] ( Figure 1). As x has a higher value, the agents move to different games. Using the same notations as in [6] that designates R (reward) when two agents cooperate, P (punishment) when two players defect, T (temptation) when one player defects and the other cooperates and S (sucker´s payoff) when one player cooperates and the other defects; the move from game to game is described in appendix D. The values of b and c also determine the transition value of x that takes the agents from one game to another. It is important to notice that the agents respond to these transitions by interacting with the others and adapt by using only simple local rules with only two basic requirements: 1) the agents must know how to distinguish that an amount is bigger than other and 2) the agents must have the notion of more is better. Within the strategies studied, other requirements are implicitly given for each case and are described in the appendix E.

ESS, RD and AD conditions
It was shown in Table 1, when the x variable reaches certain value, ESS, RD and AD behaviors emerge. Figure 2 shows the characteristic behavior obtained by the analysis described in the behavior part of the Methods section; in it, these expected behaviors are not noticeable. Analyzing the model variables, we could find that these conditions are preserved, but the effect can be observed by setting the starting population of cooperators as 2/3, 1/2 and 1/3 of the total population for ESS, RD and AD respectively and obtaining a graphic similar to Figure 2. This result is shown in appendix F along with the impact of all the parameters to each strategy.

Self-organization
Self-organization seems to be a key element for the cultural evolution of cooperation. There are several studies [8][9][6][7] [11] in which the agents just can choose between cooperate or defect. In the models presented here, the players can choose the degree of cooperation that they consider the best. Another important difference between our models and the others [6] [7] is the inclusion of a probabilistic variable so that agents choose the degree of cooperation by comparing their actual state with a past one instead of comparing their state with the state of others. This is important because each agent may be seen as an individual system adaptable to different environments or contexts using the information given by the interactions with other similar, not necessarily equal, systems.

Conclusions
Three agent-based computational models for the study of the evolution of cooperation under cultural propagation were described. It was shown that their behavior is the result of transitions between games defined by Game Theory. The transitions are consequences of the structure determined by the payoff matrixes of the three strategies studied. Each of these strategies abstracts well-known real behaviors [4]; hence the importance of creating computational models that let us experiment exhaustively different circumstances for these phenomena, a difficult task with living organisms. The impact of the parameters in each model was analyzed to better understand how to manipulate the models and adapt them for more specific studies, such as social phenomena (e.g. bulling, market formation, and migration), traffic problems (e.g. signaling issues, conflicting use of roads) and biological processes (e.g. interactions between organisms and populations).

Appendixes
A: General behavior of the agents.
program Iteration (p i ) {It must be guaranteed that each agent has followed the last instruction, then the next one may be executed} var cooperate(p i ): Boolean ; var p j : Agent ; begin 1 rotate left (random (50)); 2 rotate right (random(50)); 3 move forward number of steps(1); 4 find neighbors at distance (1); 5 p j := engage ;{this procedure are going to return the game partner of agent p i } 6 play ; 7 get consequences ; 8 tune cooperation probability ; end.
The lines 1-3 let the agents set a straight line direction but with some noise. In line 4, the agents look for other agents near them within one cell distance. The engage instruction prompts the agents to choose one neighbor (if any). Once a pair of agent engages in a game they became ineligible for other agents. The procedures in lines 6-7 contain details of each strategy.

Kin selection
Under this strategy the agents, engaged in a game, participate in a naive way supposing that they are involved in some kind of relationship. Although a player may be a cooperator, there are going to be conditions under which she will not cooperate. For example, she might not be in the mood or she isn't fast enough to do the action required by the other agent. The random decision based on the variable cp permits to simulate this and also how an agent overlooks these conditions while her cooperator attitude gets stronger (cp gets higher values). Once the players have simultaneously defined their actions, they move to the get consequences phase in which they add or subtract units from their fitness following the upper section of Table 1.
The tune cooperation probability phase is explained later in its own section.

Direct Reciprocity
In [5] a detailed explanation of the agents is given. There are two types of agents: a) agents following an always defect strategy (defectors) and b) agents following titfor-tat strategy (cooperators) that will start with cooperation. Technically, a Boolean array may be added to each agent for the tit-for-tat strategy implementation so, on the array of a player p i the last action of an agent p j will be recorded. To analyze the cultural propagation of cooperation, adjustments to these rules were made and the behavior is the following: For the get consequences phase, once the players have simultaneously defined their actions they add or subtract units form their fitness following the middle section of Table 1.

Indirect reciprocity
The development of the games under this strategy is also specified in [5]. Here the q value has a more important role. Before making a decision, each player has the probability of knowing if his game partner is a defector or a cooperator. A defector will never cooperate, while a cooperator will always cooperate if his partner is a cooperator and will not cooperate if the partner is a defector. A cooperator will cooperate with a defector with a probability 1-q, simulating the fact that the cooperator couldn't recognize the other as a defector. To study the cultural evolution of cooperation, the game is structured like the next code: When the players have simultaneously defined their actions, the behavior of the get consequences phase is to add or subtract units form their fitness following the lower part of Table 1.

C: Detailed methodology.
The size of the space in which agents interacted was thirteen by thirteen cells with 1:1 agent-cell proportion and each agent step was proportioned to 1:1 cell length. The space has no boundaries so it has a toroidal shape.
Each strategy was treated with similar analyzes with few differences in sampling. The analyzes were constructed with the idea of studying the global system in stable conditions thus it is similar to the one described in [3] and consist of simulations running 100,000 iterations outputting the average percentage of cooperators in the population. The averages were taken for the last 5,000 iterations. A simulation was made for each possible configuration described in the following sections: ─ Payoff table behavior: as it can be seen in Table 1, each strategy is dependent on three variables: b, c, and depending on the strategy, r, w, or q. Variables b and c were fixed b=4 and c=2 arbitrarily, while r, w or q (that from now on will be denoted as x) took values from the range [0.01 , 1] for kin selection (KS) and indirect reciprocity (IR) strategies and from [0.01 , 0.99] for direct reciprocity (DR). In each case, samples were taken every 0.01 steps in the x's range. w=1 was omitted to avoid division by zero.
─ Tuning criterion (TC): as explained in the Model section, after each game the agents try to adapt their behavior to the environment (tune cooperation probability). There are two criteria to do so: one based on the agent's fitness (sf) and another one based on the agent's profit (sp). For this part of the analysis, each parameter was arbitrarily fixed to the values described in Table 2.  ─ Population: in this part the impact of the number of players was explored. It was expected that there would be no impact for KS and IR but, in DR the population should be an important issue. The parameters used are described in Table 4. Robustness: as the name would suggest, the idea behind this test was to know the robustness of the strategy by varying the initial proportion of cooperators. The initial proportions were [0, 0.333, 0.5, 0.666, 1], complementary initial proportion of defectors were fixed to get the population discussed in the population section. The rest of the parameters were fixed as Table 5 details. ─ Behavior: finally average behaviors were measured. The parameters were fixed as showed in Table 6 and the results are the averages of the outputs of 10 simulations for every sample interval.  E: Implicit requirements for each game.

Kin selection
The additional condition is: agents must be able to determining the probability of genetic relatedness to other agents. In nature this is frequently observed, for example, a tiger is able of realize that it is in the territory of another tiger by the smell. Independently of the actual mechanisms used by living systems for determining genetic relatedness, we have found that this strategy is very robust. Independently of the initial proportion of cooperators in the population, if b/c > 1/r then, cooperators and defectors can coexist. A particular question may arise: if kin selection is so robust for the evolution of cooperation, why cooperative behaviors do not always occur among individuals of the same species, such as bulling between students or the same tiger of the last example fighting for territory with another tiger? One possible answer is that the ability to estimate relatedness depends on the individuals, but the perceptions of b and c are codependent of the environment and of the subjective perception of individuals. A clear and sad example of this is: the polar bears, when a female polar bear has cubs and there have been no food for several days, the mother can commit cannibalism. In this case the genetic relatedness perception remains the same (r=1) but the lack of food increases perception of the cost value for the mother and a deplorable non-cooperative consequence occurs.

Direct reciprocity
At a first glance, the additional condition of this case seems to be the ability of agents to know the probability that they have of playing again with another particular agent. Analyzing this strategy, it may be noticed that the results of this case were those with the highest error rate compared with the expected behavior. The most reasonable explanation is that, somehow, the agents notice the difference between the given w (set as a parameter) and the real one (given by the number of players in the area). One possibility is that agents attend so much to the given w with wrong results that they stop paying attention to it, although this requires further analysis.

Indirect reciprocity
The additional condition here is the ability of the agents to estimate in which proportion they actions are known by the rest of the population. This ability is only seen in organisms with complex social interactions. The analysis conducted showed that the global behavior of this case is sensitive to the initial probabilities for cooperation of the players; it was necessary to fix initial cp of cooperator=0.99 and initial cp of defectors=0.5 for the model to show the expected behavior (ESS behavior under b/c > 1/x conditions). When icpc set lower or icpd is set higher, the point in the x's range in which the cooperating population may survive is displaced to the right, thus, more players knowing the actions of other players are required for cooperation to evolve. This may be understood as the effect of a cultural stance to a particular subject. For example, corruption: if the population has cultural aspects that favor corruption, then the initial probability of cooperators and defectors will have a low value.