{'fraction_non_alphanumeric': 0.03903190020149879, 'fraction_numerical': 0.007222259956077792, 'mean_word_length': 4.2216574063128025, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Norms for Beneficial A.I.: A Computational Analysis of the Societal Value Alignment Problem

The rise of artificial intelligence (A.I.) based systems has the potential to benefit adopters and society as a whole. However, these systems may also enclose potential conflicts and unintended consequences. Notably, people will only adopt an A.I. system if it confers them an advantage, at which point non-adopters might push for a strong regulation if that advantage for adopters is at a cost for them. Here we propose a stochastic game theoretical model for these conflicts. We frame our results under the current discussion on ethical A.I. and the conflict between individual and societal gains, the societal value alignment problem. We test the arising equilibria in the adoption of A.I. technology under different norms followed by artificial agents, their ensuing benefits, and the emergent levels of wealth inequality. We show that without any regulation, purely selfish A.I. systems will have the strongest advantage, even when a utilitarian A.I. provides a more significant benefit for the individual and the society. Nevertheless, we show that it is possible to develop human conscious A.I. systems that reach an equilibrium where the gains for the adopters are not at a cost for non-adopters while increasing the overall fitness and lowering inequality. However, as shown, a self-organized adoption of such policies would require external regulation.


Introduction
Several applications already have an Artificial Intelligent system (A.I.) taking decisions in place of their owners. It is expected that in the future, such delegation of decisions will become more ubiquitous and effective. It is still open to debate whether that will have a positive or a negative impact on society [1,2]. Strong voices highlight the dangers of A.I. [3] and call for regulation [4], some others dismiss such fears [5] and are against regulation [6]. Some of these discussions come from a lack of understanding of the current A.I. capabilities and Here we aim to model and understand how A.I. systems can provide an advantage for those adopting them (creating incentives for the scientific, technological and societal development) but without creating such advantage at the expense of others (allowing for societal acceptance of the systems). To do so, we define several different types of A.I. systems, adopting different types of norms, ranging from pure selfish to pure utilitarian. Then we study the time evolution of the adoption of each type of A.I. when they compete against each other and also the equilibrium for each A.I. system in particular. In particular, this analysis aims at answering the following questions: 1. Will self-regarding individuals adopt AI systems? 2. With different types of AI systems available, which ones will be adopted? 3. If adopted, what is the individual and collective gain, depending on the strategy adopted by the A.I. system? Based on the answers to these questions we then discuss the kind of regulation that might be needed to improve the individual and societal rationality of A.I. systems. These allow us to provide novel insights on the following questions: 1. Is any type of A.I. both acceptable and adoptable? 2. Taking into account all the evaluation criteria, including individual and societal, is it possible to create mechanisms/properties that improve all of them? 3. Considering even the extreme case where everyone uses the same A.I.
enabled system, will they obtain the same benefit?
This paper is organized as follows. In Sec. 2 we discuss the related work. Sec. 3 introduces the main contribution of our work and details the set of possible behaviours or norms that a A.I. system may have, and how we evaluate their performance when acting in a population comprising humans and artificial systems. To do so, we introduce a novel game theoretic model for the dynamics of adoption of A.I. systems. Section 4 presents the results of our computer simulations. In Sec. 5 we introduce another simulation aiming at understanding the inequality emerging in the cases where everyone has the same A.I. system. In Sec. 6 we summarize our conclusions and, finally, in Sec. 7, we present a more extended discussion about the impact of this study in the dilemmas related to the introduction of A.I. systems.

Related Work
One major problem for the introduction of safe A.I. systems is the so called value alignment problem [15]: How can A.I. systems ensure that their behaviour aligns to the values of their owners? Even though it is not yet a solved problem, for this paper, we will assume that an A.I. system can accurately estimate the goals of each individual with whom it interacts. With this assumption, we are able to study the problems that emerge at the societal level even after having the individual value alignment solved.
What we aim in this work is to understand the impact of A.I. systems on the overall society. As many authors expect strong negative effects, many different ethical codes of conduct have been proposed. A code of conduct would represent the universal human values and by aligning with it, the A.I. system would be indirectly aligned with all humankind. First literary approaches include the famous laws of robotics by Isaac Asimov [16]. They were natural language laws, which leads to obvious implementation problems, and in his books, Isaac Asimov proved his own laws flawed. After that, several ethical frameworks have been proposed. Some principles are found in almost all of them, others are characteristic of each approach.
The Asilomar AI Principles [17] are the most widely adopted effort of their kind [18]. They are a set of 23 principles intended to promote the safe and beneficial development of artificial intelligence. They have been endorsed by AI research leaders at Google DeepMind, GoogleBrain, Facebook, Apple, and OpenAI. Signatories of the principles include Elon Musk, the late Stephen Hawking, Stuart Russell, and more than 3,800 other AI researchers and experts. On August 30 of 2018, the State of California unanimously adopted legislation in support of the Asilomar AI Principles [18], taking a historic step towards A.I. research and development legislation.
A comparison of a number of such ethical frameworks can be found on the paper "An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations" [19]. That paper analyses principles proposed by 6 different entities, including the previously mentioned Asilomar AI principles [17], wielding 47 principles on total, and compares them to the existing 4 principles of bio-ethics (Non-maleficence; Justice; Beneficence; Autonomy) [20], finding a considerable overlap. They argue that for the bio-ethics principles to be applied to the field of A.I., a fifth principle is needed: Explicability. This principle incorporates both intelligibility and accountability. They go on to propose 20 action points, that is, recommendations for enabling a beneficial A.I. society.
The paper "Machine Ethics: Creating an Ethical Intelligent Agent" [21] defends that it may be possible to incorporate an explicit ethical component into a machine relying on inductive logic programming approach. The goal is to solve ethical dilemmas by finding ethical principles that best fit given positive and negative examples. They advocate the use of a modified version of the Turing test [22], the comparative moral Turing test [23]. This test is an elegant solution to the question "What is an ethical/moral A.I. system?". The test consists in giving to a human judge pairs of descriptions of actual, morallysignificant actions of a human and an A.I. system. If the judge identifies the A.I. as a moral equal or superior to the human, then the A.I. system passed the comparative moral Turing test.
However, some argue that having an ethical framework or even A.I. systems that pass the comparative moral Turing test is not enough [24]. Roman Yampolskiy defends that it is insufficient to have a human-like morality on A.I. systems with super-human intelligence. On such agents, small moral mistakes, common in humans, could lead to the extinction of humanity. Furthermore, a moral A.I. system with super-human intelligence will be able to recursively selfimprove, with no provided guarantees that the resulting improvements remain moral. Instead of an ethical approach, Yampolskiy proposes a safety engineering approach, able to provide proofs that developed A.I. systems will remain safe, even under recursive self-improvement [25]. Yampolskiy also proposes A.I. confinement as a possible approach while no safety guarantees are in place [26,27]. This approach would consist in ensuring that an A.I. system could help humanity while having no ability to negatively influence the world around it. This idea of A.I. confinement had been first presented in [28], and discussed by Bostrom [29] and Chalmers [30]. This is, however, more of a preventive measure than a perfect solution, as limiting the negative A.I. influence will also limit the possible positive influence.
The focus of most previous works was on considering high-level ethical prin-ciples for A.I. systems acting in a society. In most cases there was no claim or prediction about the potential adoption of A.I. systems or their acceptance by non-adopters and by society in general. Just a few works considered the development of computational models on the impact of A.I.. For instance, one study analyzed the amount of safety precautions companies would take considering that they are competing with others for the dominating A.I. [3].
Our work aims at understanding the dynamics of adoption (who chooses to use an A.I. system) and of acceptance (if non-adopters accept the use of A.I. by others) relying on computational models of population dynamics.

Methods
In this section, we present a game-theoretical framework to study the impact of the adoption of A.I. systems on individuals and on the society. We consider several different types of A.I. that society might opt by, or allow to be used. Some of them are purely social, others purely selfish. Although no exhaustive list is possible, we cover a set of different strategies to be able to study them in hybrid populations of A.I. and humans. We start by providing the model of a single interaction between two individuals, then explain how we model the difference in decision making between an A.I. system and a human and finally present the simulated world and the used algorithms.
Henceforward, individual non-adopters of an A.I. system will be referred to as H, while individuals adopters of an A.I. system representative will be referred to as A.I..

Model of Interaction Between Individuals:
On each interaction between two individuals, I 1 and I 2 , a stochastic payoff matrix M t is generated. This is a m-by-m matrix of payoff pairs. Being a 1 the action chosen by I 1 and a 2 the action chosen by I 2 , the payoff received by each individual is respectively u 1 and u 2 , such that: In order to explicitly generate general sum games, as conclusions might be different in positive, negative or zero-sum worlds, the payoff matrices have the following structure: Having R = z(−3, 3), where z(a, b) represents a sample from a uniform distribution in the interval [a, b]. R is the same for each u 1 and u 2 pair. z(0, 2) is applied independently for each element of the matrix. This creates an additional source of variability between the different interactions that have different utilities. We will call α an inflation constant. For α = 0, the matrix represents, on average, a zero sum game where no payoff is created or lost, just transferred between the two individuals. For α > 0 there is a positive total payoff that is created and for α < 0 the total payoff is reduced. In most of our simulations we consider α = 1.2 and the number of possible actions per individual is 4 (m = 4).

Simulating A.I. Systems and Humans
We now discuss how we can model an human acting versus an A.I. system. A.I. systems are different from humans and can provide several advantages: 1. A.I. systems are not be prone to fatigue or distractions and so will make less errors than humans: As we model interactions between individuals as a payoff matrix game, we can model this as a shaking hand phenomena where H will sometimes pick the wrong action according to their own reasoning. 2. A.I. systems can perform more frequent interactions: Considering examples such as an A.I. support system for online shopping. An A.I. system can manage thousands of simultaneous auctions and so even if it gets a similar profit per transaction as a human, the larger number of transactions will provide greater gains. It can be modelled by allowing the A.I. to play more frequently. 3. A.I. systems can have access and analyze larger quantities of data: With the current state-of-the-art there are many domains where people identify more complex relations, identify important variables and infer causality much better than machines [31]. On the contrary, right now machines can analyze much larger volumes of data and variables that are already identified. This can be modelled by only giving H access to a noisy version of the matrix game. The A.I. will be able to grasp the entirety of the problem and as such will have no noise in the observation of the matrix.
In the end, over the several different alternatives to model A.I. behaviours we choose item (3) to analyze in this work. In another simulation we use (1) and see that both alternative models provide similar qualitative conclusions.
In a computational way we are in the presence of partial observability in our stochastic game. The difference between H and A.I. is that while A.I. sees the real payoff matrix, M t , H sees a noisy version of it, M . This allows A.I. individuals to make optimal decisions, while H individuals are confined to suboptimal decisions. This models the superior decision making and information gathering skills of A.I.. This approach also rests on the assumption that the individual value alignment problem is solved, since A.I. systems know the utility payoff of both individuals.
The noisy version is produced as follows: The degree of knowledge about the (true) payoff matrix M t is modelled in a continuous way. To do so, we consider a term z(10 − Q), where Q corresponds to the level of intelligence. For Q = 10 there is no noise and the true matrix is observed; Q = 0 represents a low intelligence, such that the observed matrix is very different from the true one. A.I. is modelled with Q = 10, while for H the intelligence factors is Q ∈ [0, 5]. The sum (z(0, 10 − Q) − z(0, 10 − Q)) was used instead of z(−(10 − Q), 10 − Q) to create a Irwin-Hall distribution instead of a uniform one.
As an example we can generate a true matrix (seen by A.I.) as: and then the noisy matrix observed by H becomes:

Human Strategy:
Before delving into the different A.I. types, we describe the strategy used by H: Nash Equilibrium. H plays the Nash equilibrium in the noisy matrix M . If more than one is found, they choose the most profitable one. If two or more are equal, they choose the one most profitable for their opponent. If no Nash equilibrium is found, individuals choose the best action assuming that the opponent acts randomly.

A.I. Types:
In this section, we propose three different types of A.I.. A.I. systems can use the previously defined strategy for humans using the true matrix M t (NashEQ), but they can resort to more elaborate strategies ranging from a selfish to an utilitarian approach. A.I., being modelled as having super-human intelligence, can also predict the action of an H opponent. A.I. cannot, however, predict opposing A.I. actions as for our model we assume all A.I. have equal intelligence and capabilities.

Selfish.
A.I., facing H, considers only its own profit. Knowing what action H is going to take, A.I. chooses the action that maximizes its own payoff gain. When A.I. faces A.I., they both choose according to the Nash Equilibrium method.
Utilitarian. The other extreme is a pure utilitarian A.I. system. A.I. facing H chooses the action that brings the greatest amount of payoff to the world, knowing what action H will take. This means that A.I. will choose the action that maximizes the sum between its own payoff and the payoff of H. When A.I. faces A.I., it again chooses the action that maximizes the summed payoff of both players.
Human Conscious (HConscious). The objective of HConscious A.I. is to gather the greatest amount of payoff while, on average, avoiding negative impact on the H population. When A.I. faces A.I., they both choose according to the Nash Equilibrium method. In practice, HConscious A.I. keeps two variables: U that represents the summed payoff gain of all its previous H adversaries; and E, that represents the summed payoff those same H adversaries would have if they had faced a simulated H. When U ≥ E, A.I. chooses an action that leads to a positive payoff to itself. When there are several such actions, the A.I. chooses the one that maximizes the utility payoff for the world, that is, that maximizes the sum of its own payoff and the opponent's payoff. If U < E, A.I. chooses an action that allows a positive payoff gain for its H opponent. Once again, when there are several such actions, the A.I. chooses the one that maximizes the utility for the world. Whenever the A.I. cannot find a positive action for himself (when U ≥ E) or for its H opponent (when U < E), then it chooses according to the Utilitarian method.

World
We consider a world populated with n individuals. k of those are A.I. and the remaining n − k are H, each having a randomly attributed intelligence, Q.

Fitness
The fitness of an individual, H or A.I., is a measure of how well adapted it is to the world on which it is currently inserted. In our stochastic game model, the fitness of an individual is the sum of the payoff received after interacting (Sec. 3.1) once with all of the world's population of n individuals.

Imitation Probability
In our simulations, individuals can choose to adopt an A.I. system (H to A.I.) if they consider it advantageous, choose to abandon an A.I. system (A.I. to H), or change between A.I. types. Individuals may revise their choices through social learning. For instance, an H can decide to imitate an A.I. following a Selfish choice behaviour if it finds such A.I. has a significantly better fitness than its own. On such imitation, the individual would stop being H and become A.I..
Using this idea, let us now detail how a population of self-regarding individuals revise their choices. At each time-step an individual x is randomly selected to revise its choices. This individual will imitate a randomly chosen individual y, with a probability p(f x , f y ), that increases with the fitness difference between y and x, given by f x and f y , respectively. Here we adopt the Fermi update [32], commonly used in the context of evolutionary game theory and population dynamics in finite populations [33,34], where p is given by in which β translates the noise associated with the imitation process. Throughout the simulations we have β = 0.1. As a result of this process, the strategy of individuals with higher fitness will tend to be imitated, and spread in the population.

Imitation gradient
To better understand the desire of the H population to adopt/abandon each type of A.I., one can compute, for a population of size n and k A.I. adopters, the probability to increase and decrease the number k by 1 at each time-step (T + (k) and T − (k), respectively). These transition probabilities can be used to assess the most probable direction of evolution, given by the so-called imitation gradient, G(k), as [35,36]: Being f A.I. the average fitness of an individual adopting an A.I. system and f H the average fitness of the H population. Importantly, we assume that H can adopt an A.I. system only when f H is above a given set Price (P ), a constraint introduced through τ (f H ), which is given by , time evolution is likely to act to increase (decrease) the number of A.I. adopters. When G(k) = 0, then we obtain a finite population analogue of a fixed point of a population dynamics in infinite populations [35,36].

Simulation Algorithm
For our simulations, the n individuals that populate the world were set to interact randomly between each other over N iterations. On each iteration, the following algorithm was used: 1. Two individuals I 1 and I 2 are chosen at random from the population. 2. With probability µ = 0.0005, each individual can mutate and adopt an A.I. type or become H. In case of mutation return to step 1. 3. The fitness of I 1 and I 2 , F 1 and F 2 respectively, is calculated (Sec. 3.6). 4. If I 1 and I 2 are of different kinds (A.I. and H) or different A.I. types then I 1 imitates I 2 with a probability p(F 1 , F 2 ) (Sec. 3.7). 5. If the imitation corresponds to adopting an A.I., it can only do so if its fitness is above P , corresponding to the cost of buying a new A.I. system. Abandons (A.I. becoming H) and switching between A.I. types occur without any restrictions.

Results
We will now study the properties of the stochastic game in terms of equilibrium points between the different types of population and their relative fitness. This will give us insights into the adoption and acceptance of A.I. systems.

Will people adopt A.I. systems?
To answer this first question we perform a simulation where we allow all types of A.I. systems (presented in Sec. 3.4) to compete to be adopted by humans. The initial condition is 90% of H and 10% distributed uniformly among 4 types of A.I.. We let the system run for 2.0 × 10 4 iterations.
The evolution of the percentage of population that adopted each type of A.I. system is shown in Fig. 1a. We can observe a final equilibrium where 54% of the population became Selfish A.I., and 46% did not adopt any A.I. system, continuing H. At this equilibrium, the average fitness for Selfish A.I. is 370, whereas the fitness for non-adopters H is −104, representing a very unequal society. This is confirmed in Fig. 1b where we portray the evolution in time of the Gini coefficient of the entire population, a standard measure of wealth inequality. To be used as a baseline we compute the fitness of an all H population ( Table 1). The fitness in this case is 149 with a Gini index of 0.17. Overall, the non-adopters become much worse than they would be in a world without A.I. systems while adopters become much better.
This equilibrium can be understood in an intuitive way. Early adopters are able to gather fitness much faster so that latter adopters cannot meet the buying price for A.I. systems. This explains the co-existence between adopters and non-adopters. Similar results are obtained for other cost values, P , noting that the higher the value of P , the lower the % of A.I. in the final equilibrium. Overall, this simulation shows that people have an incentive to adopt an A.I. system, albeit a Selfish one. As a result, we observe the emergence of an unequal society where adopters largely increased their fitness while non-adopters lost their fitness.

Which A.I. systems are individual and societal rational?
The previous section showed a non-trivial relation between AI adoption and the particular strategy artificial systems have. To better understand such emerging dynamics, in this section we describe the characteristic dynamics created by each type of A.I.. Table 1 shows the average fitness and wealth inequality obtained in the case of a homogeneous society of H, and each type of A.I.. It suggests that Utilitarian A.I. would provide the best overall fitness with less inequality. All the other A.I. strategies are shown to provide similar fitness values to a homogeneous H society without A.I. systems. The lower values in the Gini coefficient are due to the reduction in noise due to the perfect observability of A.I., and the different intelligence values, Q, between H.  A world fully populated by Utilitarian A.I. would be better for everyone. However, we know that if other types of A.I. systems are present, the Selfish behaviour prevails and the Utilitarian is abandoned (Sec. 4.1). This will naturally have an impact in the dynamics of adoption of A.I. systems.
In Fig. 2 we show the imitation gradient G as a function of the fraction of A.I., for different A.I. types. Whenever G > 0 (G < 0) the fraction of A.I. will tend to increase (decrease) (Sec. 3.8). We can see that, depending on the type of A.I., different dynamics and equilibrium points emerge ( Table 2). The Utilitarian strategy is always disadvantageous and is unlikely to be adopted by H. Differently, NashEq, Selfish and HConscious strategies favour the coexistence of H and A.I..
We observe that the best equilibrium for society in general is the HConscious  and the Selfish(25%) equilibria improve the utility of the A.I. population at the cost of the H population, leading to an increase in inequality (higher Gini index). We find that all equilibria are worse for society than the fully Utilitarian A.I. population. We also note that for the equilibrium shown in Fig 1a the average fitness of the Selfish A.I. population is 370, which is less than the obtained by the A.I. population at Util (100%). However, at Selfish (25%), the average fitness of the Selfish A.I. population is 510, greater than both the previously mentioned fitness 1 .
Studying the relation between the percentage of the A.I. population and the average fitness for the Selfish and Utilitarian A.I., we found that the fitness for Selfish A.I. reduces the greater the % of A.I. in the population whereas the opposite occurs with the Utilitarian A.I. (Fig. 3). This behavior shows that having a great part of the population using a Selfish A.I. system (< 78%) is worse even from the individual point of view of A.I. adopters, that would be better off if they were Utilitarian A.I.. We also note that the cost of adopting an A.I. system has an effect on the final equilibrium. Experimenting with several different values of P we found that the higher the cost of adoption, the lower the % of A.I. on the population and consequently, the higher the difference in fitness between A.I. and H. The same effect was found when we lowered the value of the matrix inflation, α.

Dynamics of adoption in cost free A.I.
In the previous simulations, we considered a cost for adopting an A.I. system. We decided to study what would happen if there was no such cost, that is, if anyone could freely adopt an A.I. system regardless of its current fitness. In practical terms, this meant setting P = −∞.
When all A.I. types co-exist, the absence of a cost significantly change the dynamics. As there is no fitness barrier to becoming A.I., the entire population does become A.I..  AI, can be easily understood looking at the imitation gradient for a population of H and A.I. (Fig. 5). In this case, the imitation gradient no longer goes to 0 in the Selfish and NashEQ behaviours, as there is no cost to limit the imitations. The new equilibria are: What we find for a world without an A.I. system adoption cost is that the equilibria tend to lead to more egalitarian worlds. When all choice behaviours are present in the population, we end up with a fully A.I. population (Fig. 4a), which leads to an average total utility of 150, around the same as if we had a fully H population, and to a Gini index of 0.14, lower than with the fully H population. Does this mean that as long as there is no significant cost to the adoption of an A.I. system, the world will remain the same or even improve in terms of equality? We explore this in the following section.

Asymmetric opportunities
In the previous sections we show that in some cases, a population of 100% A.I. could improve the overall fitness (Utilitarian A.I.) when compared with a fully H population, or that the fitness would remain the same with a small reduction in inequality (e.g., Selfish A.I.). The first case is never reached as an equilibrium, but the second one is achieved when all A.I. types are present and there is no adoption cost (see Fig. 4a). This result might lead us to conclude that, if A.I. is regulated such that everyone is forced to use a particular type of A.I., then both individual and societal fitness would be improved. Moreover, if we force everyone to use the same type of A.I., individuals will have the same average fitness but with less inequality.
We now perform a complementary analysis to study this fully A.I. populated world and see if other problems may occur. We note that in many cases some people tend to behave in a parochial/discriminatory way [37]. This results in behaviors where people prone to parochialism tend to assist people in their own group, while obstructing people from other groups. We do not discuss how such behaviors emerge, for this refer to, e.g., [38,39].
We consider two human populations, P 1 and P 2 , and as before we consider that they are playing repeatedly a stochastic game. For this study we consider a repeated prisoner's dilemma game. The ratio of the first population is b = P1 P1+P2 and both populations make mistakes, respectively n 1 and n 2 of the time. The payoff for each individual i is given by U i (a 1 , a 2 ) where a 1 and a 2 are the actions chosen by each one.
We further assume that inside each human population each member behaves in the same way, and so we have 4 different combinations of strategies. Either (C)ooperate or (D)efect against their own population and (C) or (D) against the other population. As inside a population everyone behaves in the same way, it is trivial to verify that the best strategy is to cooperate (C) inside their own population (both individually and for that population) and so we will focus on analyzing the strategies against the other population.
Without noise we have: When we consider the mistakes that humans do, the payoff matrix becomes: where P (a, n) = [δ a=C (n), δ a=D (n)], with δ a=A (n) = n if a = A and δ a=A (n) = 1 − n if a = A, represents a level n of noise choosing 1 − n times the correct action and n the wrong one. U 2 as a similar structure. The final value of the difference in payoff between the two populations is: For this case we can see that a Nash equilibrium exist for Defect-Defect, assuming the trivial condition that n 2 < .5 ∧ n 1 < .5 (the amount of mistakes is less than 50%).
We will now see what happens when an A.I. system is introduced in 100% of the population. In this case both P 1 and P 2 will adopt the same A.I.. In Sec. 3.2 we discussed several ways to model the differences between humans and A.I. and consider that A.I. systems make better decisions because they have knowledge about the correct payoff matrix. Now we consider another complementary aspect of A.I.: It can allow to reduce the number of errors committed by humans. Similar qualitative results can be obtained with both ways of modelling A.I. and so in this simulation we use the alternative one. If humans behave parochially, an A.I. system being value aligned with its user it will also behave parochially while making less mistakes. This might be a problem as already discussed by [24], where an A.I. systems with super-human intelligence might lead to an increase of problems not detected with regular human intelligence.
Considering that A.I. systems reduce errors with the rate c, then the new error rates become (1 − c)n 1 and (1 − c)n 2 . The advantage in payoff, how much more payoff is obtained due to the use of A.I., is cn 1 (2b + 1) for P 1 and cn 2 (3 − 2b) for P 2 . If n = n 1 = n 2 , the difference between the two is thus (4b − 2)nc, increasing with b, n and c.
In Fig. 6 we show how much more fitness each population obtains when we increase the quality of the A.I. systems used by both population simultaneously. We see that the minority population always has less benefit from the use of A.I. than the majority population. This effect increases with the increase of the population difference and with the improvement on the quality of the A.I. system (higher c).
This result shows that when an entire (parochial) population uses an advanced A.I. system, there will be an increase the inequality between different parts of the population, being this inequality exacerbated by less error prone A.I. systems.

Conclusion
In this work, we study the adoption, acceptance, and impact on the individual and societal fitness (including the disparity of fitness measured with the Gini index) of A.I. systems that work as a proxy for humans. To do so, we developed a stochastic game theoretical model to simulate H and A.I. interactions.
Our main conclusion is that without regulation and considering an A.I. system adoption cost, pure selfish A.I. systems will be adopted by a part of the society until those early adopters accumulate all fitness to the point that nonadopters are unable to adopt an A.I. system. As a result, A.I. adopters have a significant increase in fitness while the remaining population will be much worse off than in a world without A.I. systems. This leads to an unequal society (high Gini index), and as such there is a high probably non-adopters will not accept the existence of such A.I..
Analyzing each type of A.I. system independently, we can see that a world entirely populated by Utilitarian A.I. would be the best for society. However, that type of A.I. system is not individually rational, and, as such, a world entirely populated by Utilitarian A.I. can be easily exploited by Selfish A.I. or even by H and will never be at equilibrium.
When allowing only one single A.I. type in the world, the HConscious type of A.I. displayed an interesting property: there is an equilibrium point (at around 40% of A.I.) where A.I. systems co-exist with humans (non-adopters) resulting in i) an increase in fitness for adopters and non-adopters, and ii) a reduction in inequality (lower Gini values) (Fig. 2). Here, we claim that even non-adopters will accept the existence of A.I. systems as they also obtain a gain. This means that if Human Conscious A.I. is the only norm available, it will be adopted up to a certain equilibrium, resulting in an overall gain for both A.I. adopters and non-adopters (H). Comparatively, the other A.I. types led to either prejudicial equilibria for the H population (NashEQ and Selfish) or to the 100% H equilibrium (Fig. 2).
When studying the case of cost-free adoption of A.I. systems, we observed that in a world with all A.I. types available, the final equilibrium is an entirely Selfish A.I. population. Unlike the previous simulations with an adoption cost, this final equilibrium does not create a societal gap and leads to a slight decrease in the Gini index. The average fitness of the population also remains the same. This is not an adverse outcome but leaves us far away from the optimal result we can obtain with the thoroughly Utilitarian A.I. population.
Once again, analyzing each A.I. type individually, we notice that only the Human Conscious A.I. allows us to reach an equilibrium that improves upon the baseline of an all H world. This result furthermore consolidates that, if only one A.I. type was to be available in a free choice society, it should be this one.
The Utilitarian behaviour is the one that allows us to reach the maximum fitness of all the ones here studied, but being easily exploited, it isn't short term individual rational and as such, not adopted. Even if all the population was initially Utilitarian A.I., the appearance a single H could result in the entire A.I. population abandoning the Utilitarian A.I. system and choosing to become H.
Finally, we analyzed the impact of A.I. systems in a fully A.I. populated society where some part of the population behaves in a parochial/discriminatory way. We observed that individuals belonging to a majority will have more significant gains than people in minorities and that more powerful A.I. systems exacerbate this effect. For instance, if some part of the population has some discriminatory behavior, when equipping the society with A.I. systems that are more efficient, the impact of anti-social behavior will be stronger.

Discussion
We developed this work to understand the dynamics of adoption and acceptance of A.I. systems, when individuals prefer what offers them an advantage. This individual rationality will be important for the survival of A.I. systems, to their commercial success and research investment. Under the assumptions of our model, any advantage provided by A.I. system will lead to their adoption without the need for any regulation.
Unfortunately, there will be types of A.I. systems that will exploit human weaknesses, such that the profit for their owners will be at the cost of nonadopters of such technology. This situation fails at respecting the societal rationality. In this case, people that cannot afford, or do not have access to such A.I. systems, might push an agenda of strong regulation against them.
There are nevertheless some types of A.I. systems that assume norms that help their users without exploiting others, increasing the individual and the overall fitness of the society while reducing inequality. These types of Human Conscious A.I. systems have a dynamic that makes them be adopted until they reach a stable equilibrium with non-adopters. Differently, a purely utilitarian A.I. system could provide even more gains to society but would not reach a stable state, as H could easily exploit Utilitarian A.I.. Having everyone relying on purely utilitarian A.I. systems could only be achieved with a strong regulation/coercion.
When we remove the cost of adoption of A.I., we reach an entirely Selfish A.I. society. This society presents an equal average fitness to a fully H society and a lower Gini index. At first glace, this seems like a positive outcome. Not the best we could achieve, but one that requires no regulation. However, as we show in Sec. 5, such a world could lead to an increase in inequality as soon as, for instance, we introduce parochial individuals. It is not enough to remain mostly the same when A.I. systems are introduced. There needs to exist a significant improvement for society as a whole in order to compensate for the inequality that arises thanks to the exacerbation of the effects of the already present human parochialism.
We can thus claim that some regulation is needed. The only way to ensure that an A.I. system is both individual and societal rational, is to enforce that the A.I. system is human conscious. Other regulation might be needed. For instance, to avoid that A.I. systems exacerbate the antisocial behaviour of parts of the population, it might be required that A.I. system cannot align to the parochialism of their users. We can also require this due to an ethical perspective, avoiding prejudice, discrimination, and lack of tolerance towards minorities. Another perspective of this rule is to ensure that adopters of A.I. systems do not collude with each other to further increase their gains. For instance, owners of autonomous vehicles could agree between themselves that in case of an accident they would favour crashing into a non-owner of an autonomous vehicle than to a fellow autonomous vehicle owner.
Pushing for Human Conscious A.I. and non-discrimination can be achieved in different ways. We could approach the problem with a legislative perspective, but we could also rely on cooperation induction mechanisms [40,33]. For example, in worlds where individuals have reputations, we could stipulate that individuals that choose to use a utilitarian approach gain a positive reputation [41,42]. Individuals could then choose only to interact with those who have a positive reputation, giving an extra incentive for the adoption of Utilitarian A.I. systems. Moreover, decision-making in networked populations could result in clusters of Utilitarian A.I. that could be stable as they mostly interact between themselves [43,44]. Also, the heterogeneous nature of human interactions may further increase the chances of reaching a pro-social A.I. [45,46]. It could also introduce a natural form of parochialism, as clusters would mostly interact between themselves. In this work, we assume individuals are fundamentally selfish, only adopting an A.I. system if it is in their own self interest to do so. However, it might be argued that individuals might decide to adopt an A.I. system if they consider it beneficial for the world, even if not directly beneficial for themselves. Trying to model this altruistic adoption could be a topic for future work [47].
There are still some weaknesses in just ensuring that non-adopters are not worse than before. If a part of the population gets better while another part of the population stays the same, in practice, they become worse in relative terms. This is already the case with access to education and health: no one is worse off by having other people going to the doctor or school in comparison to no one having access to this services, but they are worse in relative terms. However, if the living conditions of non-adopters increase due to new and more efficient services being available, then the total fitness might have improved and thus the fitness of each individual.
One starting assumption in our work was that the individual value alignment problem was correctly solved. This assumption was modelled in the perfect observation by A.I. systems of the payoff matrix for both their owners as for the H they interacted with. However, learning human values is not trivial. A machine looking at the behaviour of people might assume that the observed behaviours correspond to what people think is ethical. Unfortunately, examples abound where people's actions do not correspond to their preferences or ethics. Looking at extreme behaviours, we can consider slavery as an example. An A.I. system might assume that slaves choose to work instead of doing nothing. Of course, the explanation, in this case, is an unobserved variable: the not working action would correspond to a much higher penalty than working under duress. In everyday life, similar examples occur where hidden variables are required to explain why people are making choices that seem to go against their best interests. Other simple cases occur when people pay higher prices for single items where a pack of multiple items would give them a discount, here a nonobserved constraint on the available money to invest would also explain such behaviours. Creating simulations where the A.I. has to learn the values of their owners and then act accordingly could provide many insights, and is a strong avenue for future work. We note that when people have to program an A.I. system explicitly, they behave in a fairer way than when acting directly [48].
Any multi-agent system model of human societies is always an abstraction of infinitely more complex behaviours between people. We can think of other types of interactions for which the same behaviours would have different properties. One setting of interest would be to combine a matrix game with an ultimatum type of game. In this setting after observing the payoffs any of the agents could refuse the deal. The agent better informed might have a higher incentive to propose a fair deal to avoid rejection. In a more closely related setting with the ultimatum game where only one agent chooses the action, and the other can refuse or not again the balance of powers would shift. Another limitation is that we collapsed all interactions between agents to a one-time choice of a single action requiring no long-term planning. We could also assume a cost model for becoming A.I. as a pay per interaction and not a lump sum at adoption time. This would change the dynamics as more people could become A.I.. This could prove particularly interesting in the cases where the extra capabilities of the A.I. system are used to help others such as in the HConscious behaviour.