{'fraction_non_alphanumeric': 0.0337157425111536, 'fraction_numerical': 0.008795411089866157, 'mean_word_length': 4.693396226415095, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty

In stochastic games with incomplete information, the uncertainty is evoked by the lack of knowledge about a player's own and the other players' types, i.e. the utility function and the policy space, and also the inherent stochasticity of different players' interactions. In existing literature, the risk in stochastic games has been studied in terms of the inherent uncertainty evoked by the variability of transitions and actions. In this work, we instead focus on the risk associated with the \textit{uncertainty over types}. We contrast this with the multi-agent reinforcement learning framework where the other agents have fixed stationary policies and investigate risk-sensitiveness due to the uncertainty about the other agents' adaptive policies. We propose risk-sensitive versions of existing algorithms proposed for risk-neutral stochastic games, such as Iterated Best Response (IBR), Fictitious Play (FP) and a general multi-objective gradient approach using dual ascent (DAPG). Our experimental analysis shows that risk-sensitive DAPG performs better than competing algorithms for both social welfare and general-sum stochastic games.


INTRODUCTION
A Bayesian game [13] is a game of incomplete information in which multiple players or agents act simultaneously and the corresponding uncertainty is modelled using Bayesian posterior updates. The goal of the game is to identify a set of strategies or policies that determine how each of the agents will interact with the environment while optimising the corresponding utility functions. We delineate between pure, i.e. deterministic, and mixed, i.e. stochastic, strategies. Upon fixing the set of strategies that the agents will use, the game can be evaluated and the utility of each of the players can be measured. Bayesian game formalism has been used to model threats in the defence industry [5], autonomous driving [27], intrusion detection in wireless networks [20], auctions [25] and more.
Depending on what their individual goals are, the players could act in a cooperative [26], adversarial [17] or individual manner. In the case where all of the players have identical utility functions, we call the game cooperative. If instead, the goal of all the agents is to maximise the joint utility, we term the game a social welfare game. Further, some agents could attempt to minimise the utility of others, in that case, we have an adversarial game. Finally, if there is no restriction on the utility functions, we have a general game. The flexibility of the framework of the Bayesian game allows for the optimisation of any of these goals by specifying utility functions for each of the agents. Thus, the risk-sensitive Bayesian game formulation and the corresponding algorithms that we propose are independent of the choice of the utility function. Due to brevity of space, we perform experimental evaluations only for social welfare and general games.
Related Works: Formulations. A line of research reduces multiagent reinforcement learning (MARL) to the stochastic game setting where the agents aim to maximise their own utility functions [7,15,19]. Under the assumption that all players except one have fixed stationary strategies, the problem is formulated as a partially observable Markov decision process (POMDP) and solved using traditional reinforcement learning (RL) techniques.
Mansour et al. [22] studied Bayesian games with a focus on incentivising exploration which induces risk-seeking behaviour in order to learn more about the task at hand. Tian et al. [31] investigated the risk inherent to a stochastic game and provide algorithms for risk-averse agents. The risk studied in [6,31] is more akin to aleatory, i.e. inherent, risk encountered in RL previously [23].
In contrast, our work considers the risk associated the incomplete information about types, i.e. the utility function and policy space. The risk in Bayesian games due to type uncertainty is connected to the works in RL considering epistemic risk [8,12] as the Bayesian game can be considered a multi-task RL problem where multiple agents are optimised simultaneously. In the case with only one adaptive agent, this reduces to doing epistemic risk-sensitive RL.
Related Works: Algorithms. A multitude of approaches are proposed to optimise the behaviour of the agents in Bayesian games. One of the approaches is Iterated Best Response (IBR), where the algorithm optimises every player's strategies iteratively by selecting the best response for one agent given all the other agents. Ho et al. [16] used this approach to study problems where multiple developers are to construct resorts near natural attractions, where each developer is acting egotistically. In [25], the authors apply IBR to identify Bayesian Nash equilibria in auctions. The obtained best-response strategy in this setting is identical among bidders.
Another approach is Fictitious Play (FP) [4], where one agent is playing fictitiously against the other. Typically, this is done by letting one agent play against a combination of the other agent's past behaviour. For instance, assuming the other agent's strategy is an average over all of the actions it has taken so far. Shapley [30] shows that for problems with more players and more strategies per player, as in a Bayesian game, there can exist solutions without convergence. But in cooperative setting, the fictitious play demonstrated convergence guarantees Monderer and Shapley [24]. In this work, we use a variant of FP that is a rolling average over the past weights of the other agent.
Finally, one can simply optimise all agents simultaneously by updating all of their strategies by ascending according to each of their objectives. This approach (for the two-player case) is called Dual Ascent Policy Gradient (DAPG) henceforth in this work. Multiobjective policy gradients have been studied before in the field of MARL, Lu et al. [21] propose a safe gradient descent ascent approach for multiple decentralised agents. A crucial difference between their and our work is the MARL setting, while our work is in a Bayesian game. Further, the risk studied in their work is the inherent (aleatory) risk. Another recent contribution by Kyriakis et al. [18] involve multi-objective policy gradients used to identify Pareto optimal solutions with unknown and linear utility functions for RL. The setting is notably different as our work focuses on the Bayesian game setting. Likewise, if the problems is cast as a multitask setting, something that does not apply to the general Bayesian game setting, then Sener and Koltun [29] propose an applicable solution to identify Pareto optimal solutions using gradient-based multi-objective learning.
There exist also attempts to unify these frameworks, for instance, Lanctot et al. [19] generalises previous approaches, such as IBR, FP and more to a general framework. From this, they propose a new algorithm, Policy-Space Response Oracles, which constructs a meta-strategy taking all of the players' utilities into account. Another work, by Heinrich and Silver [15] develop Neural Fictitious Self-Play, an extension of Fictitious Self-Play [14], where agents play against themselves, fictitiously. These strategies are then compared to classical RL algorithms. Contribution: We propose a novel framework for managing risk due to type uncertainty in Bayesian games. To the best of our knowledge, this has not been done before. We relate the risk that arises due to type uncertainty to the risk of not knowing the policies of the other players in MARL. Finally, we demonstrate the superiority of the framework of handling this risk experimentally by introducing three risk-sensitive versions of commonly used algorithms for solving stochastic games and evaluating them on toy problems. These are risk-sensitive iterated best response (RS-IBR), risk-sensitive fictitious play (RS-FP) and risk-sensitive dual ascent policy gradient (RS-DAPG).

BACKGROUND
A Bayesian game can be defined in a multitude of ways. In this work, we adopt the framework given by Harsanyi [13] where a set of players act in an imperfect information game with a common prior over types.
Definition 1 (Bayesian Game). A Bayesian game can be formulated as G = ( , , S, A , R × , T , , ), where is the number of players, the number of possible types, S the state space of the problem, A the action space of player , R the reward function of player for type , T : S × A → Δ(S) the transition distribution of the problem, a common prior over types and a common discount factor.
In particular, a Bayesian game G is equipped with a set of types, 1 , 2 , where each type uniquely specifies the utility function and behaviour of a player. Thusly, part of the uncertainty associated with a Bayesian game comes from each player not knowing which type they will be and what type the other players will be. An agent must then prepare itself for all possible player and type configurations and arrive at strategies that can perform well in each scenario.
To evaluate a game G, it needs to commit to a set of policies Π × , one policy per player and type combination. We denote the policies by { 1 1 , . . . , 1 , 1 2 , . . . , 2 } ∈ Π for a two-player game with types. The expected utility , 1 associated with player 1, when player 1 has type and player 2 has type , can be defined in the following way, Note that the utility associated with type for player 1 depends not only on the policy of player 2 for type but for all combinations of types. Combining all the utilities for a particular player by marginalising over the prior can be done in the following way, Definition 2 (Expected Utility of a game G). The expected utility marginalised over the prior for a given game G is

RISK IN BAYESIAN GAMES
In this work, the risk associated with the uncertainty about types is studied. That is, , combined with the utility of each of the individual players for that particular type configuration. .,. 1 and .,.
2 are thus discrete probability distributions with mass equal to ( 1 = , 2 = ) and values , 1 and , 2 . By considering all possible combinations of types we get the following probability mass function, and similarly for the other agent, 2 ( ). In this work, we consider conditional value-at-risk (CVaR), studied for instance by Artzner et al. [1], which is a risk measure that focuses on the % worstcase outcomes of the distribution. CVaR can be defined as follows, Earlier works [11,28] have shown that coherent risk measures, like CVaR, can be written as perturbed distribution functions, and expectation is taken w.r.t the perturbed distribution yields riskadjusted utilities.

ALGORITHMS
In this section, we describe the basis of the risk-sensitive algorithms proposed in this work. They are, IBR, FP and DAPG with respective risk-sensitive versions RS-IBR, RS-FP and RS-DAPG. All algorithms use softmax policies for type 1 = and player 1. ∈ R × × | S |× | A | is thus the full parameter set.

Iterated Best Response
IBR [3,16,25] is an algorithm that iteratively identifies the optimal policy for a given player, keeping the policies of all other players fixed. Since IBR also only updates a subset of the available parameters at each iteration, it might have convergence problems for the general setting. The IBR procedure can be seen in Algorithm 1.

Fictitious Play
One application of the FP [4,24,30] framework can be seen in Algorithm 1. The setup is similar to that of IBR but with the important distinction that the objective is taken w.r.t a rolling average of the previous parameters of agent one for agent two, wherē 1, +1 ≜ 1 +1 +1 =0 1, .

Dual Ascent Policy Gradient
The final algorithm uses the utilities from multiple objectives to compute a gradient step over the whole parameter set. As there are only two agents, with one objective respectively, the algorithm is termed dual ascent and is ascending simultaneously using both objectives. Note that the gradient ∇

EXPERIMENTS
We investigate randomly generated toy problems, where T is generated from a product of Dirichlet distributions, all with parameter = 1. The individual reward functions for the players and types are also randomly generated, where R is generated from a product of normal distributions with parameters = 0, = 1. Performance of the algorithms is then evaluated in two distinct settings, one where the social welfare is maximised and the other in a general multi-objective setting.
The metrics of interest in the social welfare setting is the average utility ( 1 + 2 )/2 among the players and a risk-adjusted version end for 7: end for Table 1: Social welfare setting, evaluated over 100 toy experiments. The first column indicates the algorithm evaluated. If the algorithm is prefixed by RS-, then it uses a risksensitive objective. The second column is the social welfare objective ( 1 + 2 )/2. An algorithm with higher social welfare performed better. The third column displays the riskadjusted social welfare objective, using CVaR with threshold parameter = 0.25. All results are averaged over 100 different games and the mean and standard deviation is reported.

Agent
Social Welfare ± CVaR =0. 25    In the general setting with multiple objectives, we study the individual utilities among the agents and the algorithms with the best performance is the set of non-dominated algorithms in a Pareto optimality sense.

Social Welfare
A stochastic game consisting of multiple players with a common goal to maximise the social welfare can be re-instantiated as a Bayesian MDP, which can be solved approximately using backward induction, see MMBI in Dimitrakakis [9] for the risk-neutral case and RS-MMBI in Eriksson and Dimitrakakis [12] for the risksensitive case. The results acquired from evaluating the social welfare setting is summarised in Table 1.

General Case
In the general case, there are multiple objectives, one for each agent. We, therefore, study the concept of Pareto optimality [2] to determine which algorithms produce the best results. Figure 1 displays the Pareto front by the non-shaded region. The points in the shaded region are all dominated by points in the non-shaded region. The Pareto front of dominating algorithms, (A), can be defined as, that is, the set of non-dominated algorithms in A. The Pareto fronts of the risk-neutral and risk-sensitive objectives as seen in Figure 1 seem to indicate the general gradient approach in Algorithm 2 is superior to the IBR and FP approaches. In Table 2 we can see that the risk-sensitive algorithms outperform the risk-neutral algorithms when optimising for the risk-sensitive objective as one would expect. Likewise, for the risk-neutral objective, the riskneutral algorithms outperform. We can see that the pure strategies generated by MMBI and RS-MMBI perform well, although they are bested by the dual ascent approach in both settings.

DISCUSSION AND FUTURE WORK
Extensions of this work would include evaluation of the algorithms in the cooperative settings described in Dimitrakakis et al. [10], they are, Multilane Highway and Food and Shelter which would be interesting testbeds for the algorithms. Another setting to consider is given by Tian et al. [31] which is a navigation task evaluated for risk-sensitive games. In their work, they also describe another interesting risk measure.
Proper convergence analysis of the dual ascent policy gradient algorithm for Bayesian games is also something worth looking into as well as the identification of the Pareto optimal set of solutions in order to study Bayesian Nash equilibria in this setting.