{'fraction_non_alphanumeric': 0.048814188959612204, 'fraction_numerical': 0.01082746279475389, 'mean_word_length': 4.510234648027958, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bounded Risk-Sensitive Markov Game and Its Inverse Reward Learning Problem

Classical game-theoretic approaches for multi-agent systems in both the forward policy learning/design problem and the inverse reward learning problem often make strong rationality assumptions: agents are perfectly rational expected utility maximizers. Specifically, the agents are risk-neutral to all uncertainties, maximize their expected rewards, and have unlimited computation resources to explore such policies. Such assumptions, however, substantially mismatch with many observed humans' behaviors such as satisficing with sub-optimal policies, risk-seeking and loss-aversion decisions. In this paper, we investigate the problem of bounded risk-sensitive Markov Game (BRSMG) and its inverse reward learning problem. Instead of assuming unlimited computation resources, we consider the influence of bounded intelligence by exploiting iterative reasoning models in BRSMG. Instead of assuming agents maximize their expected utilities (a risk-neutral measure), we consider the impact of risk-sensitive measures such as the cumulative prospect theory. Convergence analysis of BRSMG for both the forward policy learning and the inverse reward learning are established. The proposed forward policy learning and inverse reward learning algorithms in BRSMG are validated through a navigation scenario. Simulation results show that the behaviors of agents in BRSMG demonstrate both risk-averse and risk-seeking phenomena, which are consistent with observations from humans. Moreover, in the inverse reward learning task, the proposed bounded risk-sensitive inverse learning algorithm outperforms the baseline risk-neutral inverse learning algorithm.


Introduction
Markov Game (MG), as an approach to model interactions and decision-making processes of intelligent agents in multi-agent systems, dominates in many domains, from economics [1] to games [25], and to human-robot/machine interaction [3,8]. In classical MGs, all agents are assumed to be perfectly rational in obtaining their interaction policies. For instance, in a two-player Markov Game, at each step t, agent 1 is assumed to make decisions based on his/her belief in agent 2's behavioral model in which agent 2 is also assumed to behave according to his/her belief in agent 1's model . . . and both agents are maximizing their expected rewards based on the infinite levels of mutual beliefs. If the beliefs match the actual models, perfect Markov strategies of all agents may be found by solving the Markov-perfect equilibrium (MPE) of the game where a Nash equilibrium is reached. Under such assumptions, we can either explore humans' optimal strategies with pre-defined rewards (forward policy learning/design) or perform the inverse learning problem in which we aim to recover humans' reward structures by observing their behaviors based on, for instance, the maximum entropy inverse reinforcement learning [40].
However, real human behaviors often significantly deviate from such "perfectly rational" assumptions from two major aspects [9]. First, mounting evidence has shown that rather than spending a great amount of effort to hunt for the best choice, humans often choose actions that are satisfying (i.e., actions that are above their pre-defined thresholds according to certain criteria) and relatively quick and easy to find. Simon [27] formulated such a decision-making strategy as bounded rationality. Among the many developed models to capture such bounded rationality, iterative human reasoning models from behavioral game theory [4] are one of the most prominent paradigms. These models do not assume humans perform infinite layers of strategic thinking during interactions but model humans as agents with finite levels of intelligence (rationality). Second, instead of optimizing the risk-neutral expected rewards, humans demonstrate strong tendency towards risk-sensitive measures when evaluating the outcomes of their actions under uncertainties. They are risk-seeking in terms of gains and risk-averse for losses. For example, people often prefer a choice with high reward but low probability over the one with low reward but high probability although the latter has a higher expected value. Humans also have stronger motivations to avoid losses than to obtain gains [34]. Such deviations make modeling real humans' interactive behaviors using classical MGs very difficult.
In this work, we aim to establish a new game-theoretic framework, i.e., the bounded risk-sensitive Markov Game (BRSMG) framework that considers the two aspects of realistic human behaviors discussed above. The integration of bounded rationality and risk-sensitive measure have failed the well-established Nash equilibrium strategies, and fundamental questions such as the convergence of policy learning and reward learning have to be re-visited and proved. Hence, our goal is to develop general solutions to both the forward policy learning/design problem and inverse reward learning problem in general BRSMG framework.
More specifically, in the forward strategy learning/design problem, rather than introducing additional computational costs to agents' actions as in [2], we model humans' cognitive limitations and biases by adopting the iterative human reasoning models, i.e., humans' strategic reasoning is not of infinite layers, but rather iterative and finite. Moreover, to model the influence of humans' risk sensitivity, we study agents' optimal policies in terms of maximizing their cumulative prospects rather than the expected utilities according to the cumulative prospect theory (CPT) [34]. In the inverse reward learning problem, we develop a bounded risk-sensitive inverse learning algorithm which can recover not only the nominal rewards of agents but also the intelligence levels and the parameters in their risk measures directly from their demonstrated behaviors, with no prior information on their intelligence levels. To our best knowledge, our work is the first to consider both bounded rationality and risk-sensitivity in general-sum MGs for both the forward and the inverse problems. bounded risk-sensitive Markov Games: two human agents (orange and blue) plan to exit the room through specified doors without collisions with obstacles and each other. We aim to answer two questions: 1) assuming both humans have bounded level of strategic reasoning and risk-sensitive performance measures, how will their optimal policies differ from that of classical MGs? and 2) what algorithms can recover the rewards and sensitivity parameters given their trajectories?
Contribution. In summary, the contribution of our work is three-fold: 1) we proposed a novel game-theoretic framework, BRSMG, that captures the bounded rationality and risk-sensitivity in humans' reasoning processes; 2) the proposed framework makes the first attempt to establish a bridge between inverse reward learning and riskaware iterative reasoning models in behavioral game theory; 3) in contrast to the previous riskneutral reward learning algorithms that learn the nominal rewards under equilibrium solution concepts, we exploit an alternative paradigm based on non-equilibrium solution concepts and offer a solution that learns not only the rewards but also the intelligence levels and the risk-sensitive measure parameters. Therefore, our solution provides an interpretable and heterogeneous human behavior model which is of critical importance in human-robot interactions.

Related Work
Bounded rationality. The influence of bounded rationality in the forward strategy design problem has attracted researchers' attentions in both single-agent and multi-agent settings. One group of studies formulate such influence by introducing additional computation costs to agents' actions, such as [2], [11] and [12]. Another group focuses on models that can explicitly capture the bounded reasoning process of humans. Examples include the Boltzmann rationality model [35], the quantal response equilibrium solution (QRE) [21], and various iterative reasoning models such as the level-k framework [7], the cognitive hierarchy model [5], and the quantal level-k model [29] in multi-agent setting. The Boltzmann and QRE model formulate all irrational behaviors of humans via sub-optimality, while iterative reasoning models emphasize more on the reasoning depths of humans. Instead of assuming all agents perform infinite levels of (circular) strategic reasoning, iterative reasoning models assume that humans can only perform a finite number of iterations of reasoning. Applications of all the such models can be found in many domains, including normal-form zero-sum games [32], aerospace [38], cyber-physical security [15,16], and human-robot interaction applications [18,8,33]. It is proven in [36] that compared to QRE, the iterative reasoning models can achieve better performance in predicting human behaviors in simultaneous move games. More specifically, [37] showed that the quantal level-k model is the state-of-the-art among various iterative reasoning models.
Risk measure. Many risk measures have been developed to model how decisions should be made under uncertainties beyond expectation. Value-at-risk (VaR) and conditional value-at-Risk (CVaR) [22] are two well-adopted risk measures. In addition, the cumulative prospect theory (CPT) [34] also formulates a model that can well explain a substantial amount of human risk-sensitive behaviors that are beyond the scope of expected utility theory, and has been verified in many domains. In the light of the above risk measure models, many risk-sensitive decision-making and reward learning algorithms have been proposed for single-agent settings, such as [19,6,13,17] for decision-making and [20,23] for inverse reward learning. In [30], the authors studied the inverse reward learning problem in a multi-agent setting via a Stackelberg Game [26] where a leader-follower formulation (ego agent is assumed to be the leader from his/her own perspective in the game) was assumed during the risk-sensitive inverse learning process. However, such a leader-follower formulation assumes all agents as homogeneous agents and thus can not capture the diversity of humans in terms of their reasoning capabilities.
Inverse reward learning in multi-agent games. The inverse reward learning problem in multi-agent games have also recently attracted researchers' attention. [39] first adopted the quantal response equibrium concept to model the interactions among agents and developed a maximum-entropy multiagent inverse reinforcement learning algorithm. [10] further extended the idea for better efficiency and scalability by introducing a latent space in the reward network. However, the multi-agent reward learning problem with iterative reasoning models and CPT risk measure has never been addressed. In this work, we propose the BRSMG framework to fill the gap.

Classical Markov Game
In classical two-player MGs, each agent is represented by a Markov decision process (MDP). We denote a MG as G P, S, A, R, T ,γ , where P={1, 2} is the set of agents in the game, S=S 1 ×S 2 and A=A 1 × A 2 are, respectively, the joint state and action spaces of the two agents, R=(R 1 , R 2 ) is the set of agents' one-step reward functions with R i : S × A i × A −i → R, T : S × A → S represents the state transition of the game (we consider deterministic state transitions in this paper), −i = P \ {i} represents the opponent of agent i, andγ is the reward discount factor that shared by both agents.
We let π i : S → A i denote a deterministic policy of agent i. At step t, given current state s t , each agent in classic MGs is trying to find the optimal action that maximizes his/her expected total discounted rewards. Namely, the optimal policy π * ,i is given by π * ,i = arg max t+τ , a −i t+τ ) represents the value function at s t , i.e., the expected return starting from s t under policy π i with uncertainties on the estimate of the opponent's policy π −i . The notations a −i t+τ and s t+τ , respectively, represent the predicted future action of the opponent and the corresponding state at step t + τ . At the MPE, both agents achieve their optimal policies. Due to the mutual influence between the value functions of both agents, finding the MPE policies is typically of an NP-hard problem.

Quantal level-k model in iterative reasoning models
The quantal level-k model is one of the most effective iterative reasoning models in predicting human behaviors in simultaneous move games [37]. It assumes that each human agent has an intelligence level that defines his/her reasoning capability. More specifically, the level-0 agents do not perform any strategic reasoning, while agents with higher levels of intelligence (k ≥ 1) make strategic decisions by treating other agents as quantal level-(k−1) agents and forming their quantal best response to the predicted quantal level-(k−1) policies. As shown in Fig. 1, the orange agent is a level-1 agent and he believes that the blue agent is a level-0 agent. Meanwhile, the blue agent, which is in fact a level-2 agent, believes the orange one as a level-1 agent while moving. The quantal level-k model has therefore reduced the complex circular strategic thinking in classical MGs to finite levels of iterative optimizations. With an anchoring level-0 policy, policies under different rationality levels, namely, k = 1, 2, 3, . . . , can be sequentially and iteratively solved for each agent. Moreover, experiment results in the p-beauty contest game found that the average intelligence levels for human is k = 1.5, and over 80% people do, at most, two reasoning steps, i.e., k max = 2 [28].

Cumulative prospect theory
The cumulative prospect theory (CPT) is a non-expected utility theory proposed by Kahneman and Tversky in [14] to describe the risk-sensitivity of humans' decision-making processes. It can explain many systematic biases of human behaviors deviating from risk-neutral decisions such as risk-avoiding, risk-seeking, and framing effects [14].
Definition 1 (CPT value). For a random variable X, the CPT value of X is defined by 1. If X is continuous, then 2. If X is discrete satisfying n i=−m P(X=xi)=1, xi≥x 0 for i=0, · · ·, n, and xi<x 0 for i=−m, · · ·, −1 , then They describe the characteristics of humans to deflate high probabilities and inflate low probabilities. The two functions u + : R→R + and u − : R→R + are concave utility functions which are, respectively, monotonically non-decreasing and non-increasing. The notation x 0 denotes a "reference point" that separates the value X into gains (X≥x 0 ) and losses (X<x 0 ). Handling gains and losses separately is a key feature of the CPT model and it captures the different preferences of humans towards gains and losses. Moreover, the slope of u − is usually larger than that of u + to show that humans weigh losses more than gains. Without loss of generality, we set x 0 = 0 and omit x 0 in the rest of this paper. Note that when both the probability weighting functions and the utility functions take the identity function, i.e., (1) and (2) showing the connection to the expected value, i.e., the risk-neutral performance measure.
Many experimental studies have shown that representative functional forms for u and w are: The parameters α, β, γ, δ∈(0, 1] are model parameters. We adopt these two representative functions in this paper, and focus on the discretedomain of CPT value in (2). Section 1 of the supplementary material illustrates the probability weighting functions and the utility functions.

Bounded Risk-Sensitive Markov Game
As discussed in Section 1, classical MGs implicitly assume: 1) all agents are risk-neutral, i.e., they are maximizing their expected utilities under uncertainties, and 2) all agents are unbounded in terms of their intelligence levels and computation resources. In this section, we investigate the agents' policies in a new general-sum two-player MG, where each agent is bounded-rational with a risk-sensitive performance measure.

Bounded risk-sensitive policies
According to the quantal level-k model in Section 3.2, agents in the bounded risk-sensitive Markov Games are of finite-level intelligence. Specifically, if agent i is assumed to have intelligence level-k, k ∈ N + , then he/she believes that the opponent player is of level-(k−1), and forms his/her own quantal level-k policies. To solve for the risk-sensitive quantal level-k policy π * ,i,k , we start with defining a level-0 policy as the anchoring policy. Definition 2 (The anchoring policy of level-0 agents). We define the level-0 policy as the anchoring policy. The level-0 policy is an uncertain-following policy. Namely, at time step t, given state s t and the action a −i from the opponent agent (i.e., the leader), the stochastic policy of a level-0 agent i satisfes Note that we exploit the uncertain-following policy as the level-0 policy for the purpose of illustrating the proposed framework. The selection of level-0 policy can be changed in different application domains, for instance, rule-based policies have been used as the level-0 policies in [18,33], and [31] discussed the influences of different rule-based policies for interactive planning.
With the anchoring policy defined, we can start the iterative process to solve for the optimal policies under higher levels of intelligence.
First, based on the CPT model defined in (2), given current state s t , a quantal level-k agent i tries to maximize the following discounted future cumulative prospects: where π * ,−i,k−1 : S × A −i → [0, 1] denotes the optimal risk-sensitive quantal level-(k − 1) policy of agent −i, whose level of intelligence is believed to be (k−1) from agent i's perspective. The action a −i t+τ denotes the predicted action of agent i sampled from π * ,−i,k−1 at time step t+τ . We let π * ,i,k denote the optimal risk-sensitive quantal level-k policy of agent i, and define V * ,i,k (s t ) J π * ,i,k (s t ) in (4) as the optimal CPT value at s t . It represents the optimal CPT value that agent i could collect in the future if he/she executes the optimal policy π * ,i,k . Therefore, similar to classical MGs, according to (4), the optimal CPT value at any s ∈ S satisfies [24,19]: Also, at state s, we define the optimal value of agent i induced by taking action a i as its opti- , similar to classical MGs. Based on such definitions and the Boltzmann model [35], the optimal policy π * ,i,k for all k≥1 can be written as where β≥0 defines the level of the agents conforming to the optimal strategy. We set β=1.
Next, we need to find the optimal quantal level-k risk-sensitive policy π * ,i,k by solving (5) and (6) for i ∈ P with k = 1, 2, . . . in an iterative way.

The convergence of bounded risk-sensitive policies
Theorem 1. Define the tuple s, a i , a −i :=c a −i s,a i and normalize the transformed probabilities For an arbitrary agent i∈P, if the one-step reward R i is lower-bounded by R min with R min ≥ 1, then ∀s ∈ S and all intelligence levels with k=1, 2, · · · , the dynamic programming problem in (5) can be solved by the following value iteration algorithm: Moreover, as m → ∞, V i,k m+1 converges to the optimal value function V * ,i,k (s). Proof. We prove the theorem by induction. For k=1, π * ,−i,k−1 =π −i,0 in Definition 2, and with (7), (5) reduces to a single-agent policy optimization problem where the value iteration algorithm can be proved for convergence. Then we prove that for any level k>1, π * ,−i,k−1 is known a priori and does not depend on a i , which again reduces (5) into a single-agent policy optimization problem. More details are given in Section 2 of the supplementary material. The algorithm in Theorem 1 is summarized in Algorithm 1, where the notation B denotes a CPT-based Bellman operator and is defined as:
In Section 4, we investigate the forward policy learning problem of agents in BRSMGs. In this section, we consider the inverse problem, that is, inferring agents' reward functions, their risk-sensitive parameters and their levels of intelligence from their interactive behaviors.

Problem formulation
In the inverse reward learning problem, we are given demonstrated trajectories of two interacting agents who are running the policies in BRSMGs. We do not know a priori their intelligence levels. We let x 0 =0 as the reference point in CPT, and consider the risk behaviors induced by the weighting function Moreover, we assume that the one-step rewards for both agents can be linearly parameterized by a group of selected features: Under such circumstances, we defineω=(γ,ω r ), whereγ=(γ i , γ −i ) andω r =(ω i , ω −i ), respectively, represent the parameters in the weighting functions and reward functions of both agents. Thus, given a set of demonstrated trajectories from the two players in a BRSMG denoted by , the inverse problem aims to retrieve the underlying reward parameters and risk-sensitive parameter, i.e.,ω, from D. Based on the principle of Maximum Entropy as in [40], the problem is equivalent to solving the following optimization problem: where P(ā t |s t ,ω) is the joint likelihood of agents' actions conditioned on states and parameters. We solve this optimization problem via gradient ascent. First, we need to find out its gradient.

Gradient of the log-likelihood of a demonstration
The log-likelihood of a joint trajectory ξ is given as follows: where π * ,i,k ī ω and π * ,−i,k −ī ω , respectively, represent the risk-sensitive quantal level-k policies of agent i and agent −i parameterized byω. The probability P(k|ξ t−1 ,ω), k ∈ K, is the posterior belief in an agent's intelligence level based on the joint trajectory history ξ t−1 upon time t−1. Note that initially, we set P(k|ξ −1 ,ω) as an uniform distribution on K. Therefore, P(k|ξ t−1 ,ω) can be updated recursively from t = 0 via: Intuitively, (10) and (11) mean that without prior knowledge on the intelligence level of each agent, we need to update the beliefs in the roles of both human agents simultaneously as we evaluate the likelihood of a joint trajectory, given parameter setω.

Gradients of policies
Recalling (6), ∂π * ,i,k ω /∂ω, ∀i∈P and k∈K, requires the gradient of the corresponding optimal Q function with respect toω, i.e., ∂Q * ,i,k ω /∂ω. Due to the max operator in (5), direct differentiation is not feasible. Hence, we use a smooth approximation for the max function, that is, The parameter κ>0 controls the approximation error, and when κ→∞, the approximation becomes exact. Therefore, (5) can be re-written as Taking derivative of both sides of (12) with respect toω yields (note that (·) ω := ∂(·)ω ∂ω ) We can see that in (13), V , * ,i,k ω is in a recursive format. Hence, we propose below a dynamic programming algorithm to solve for V , * ,i,k ω and Q , * ,i,k ω at all state and action pairs.
Proof. We first define ∇BV ,i,k m = V ,i,k m+1 , and show that the operator ∇B is a contraction under the given conditions (derivations of ∂ρ ī ω /∂ω are shown in Section 3 of the supplementary material). Then, the statement is proved by induction similar to Theorem 1. More details are given in Section 4 of the supplementary material.

Gradient of the posterior belief
The second gradient that we need to compute is the gradient of the posterior belief of k with respect toω, i.e., ∂ log P(k|ξ t−1 ,ω)/∂ω. Recalling the definition of the posterior belief over k in (11), we have ∂ log P(k|ξ t−1 ,ω)/∂ω depending on ∂π * ,i,k ω /∂ω(s t−1 , a i t−1 ) and ∂ log P(k|ξ t−2 ,ω)/∂ω for all k∈K. Substituting the gradients of policies developed in Section 5.2.1 yields a recursive format from time 0 to time t − 1 which can then be easily computed.

Parameter learning algorithm
Algorithm 2: Compute gradient of the bounded risk-sensitive policies Input: Markov Game model G, highest intelligence level kmax, and π * ,i,k , i ∈ P and k = 1, . . . , kmax. We summarize the value iteration algorithm that computes the policy gradient in Algorithm 2. Based on this, the gradient ascent algorithm is used to find local optimal parametersω that maximize the log-likelihood of the demonstrated joint behaviors of agents in BRSMG. The algorithm is summarized in Algorithm 3.

Experiment
In this section, we utilize a navigation example to verify the proposed algorithms in both the forward policy design and inverse reward learning problems in BRSMG. The simulation setup is shown in Fig. 1. Two human agents (blue and orange) in a room are required to exit the room through two different doors while avoiding the obstacles in the environment and potential collisions with each other. We assume that the two agents move simultaneously and there is no communication between them, but they can well observe the actions and states of each other in the previous time step. Moreover. We let k max = 2 in this experiment since psychology studies found that most humans perform at most two layers of strategic thinking [28].

Environment setup
We define the state as s=(x 1 , y 1 , x 2 , y 2 ), where x i and y i denote the coordinates of the human agent i, i∈P. The two agents share a same action set A={move left, move right, move up, move down}.
At each state, the reward of agent i includes two elements: navigation rewards and safety rewards which reflect, respectively, how close the agent is to the target door and how safe it is with respect to collisions with obstacles or the other agent. We restrict all rewards to be positive, satisfying R min =1, namely, if a collision happens, an agent will collect only a fixed reward of 1. If there is no collision, then agents can receive rewards greater than 1 according to the navigation rewards. The navigation rewards are set as shown in Fig. 2: the highest rewards will be collected if the agents reach the doors.

The foward problem: interactions in Bounded Risk-Sensitive Markov Games
In this section, we investigate the influence of the risk-sensitive performance measure to agents' policies in Markov Game by comparing the interactive behaviors of agents under risk-neutral and risk-sensitive policies. We set the parameters in the CPT model as γ 1,2 =0.5 and α 1,2 =0.7. Three cases are considered: Case 1 -both agents are quantal level-1 (L1-L1); Case 2 -both agents are quantal level-2 (L2-L2); and Case 3 -one agent is quantal level-1 and the other is quantal level-2 (L1-L2). If both agents exit the environment without collisions and dead-locks, we call it a success. We compare the rate of success (RS) of each case under risk-neutral and risk-sensitive policies in 100 simulations with agents starting from different locations.
First, let us see how a risk-neutral agent behaves under different levels of intelligence. Based on the anchoring policy (level-0) in Definition 2 in Section 4.1, a risk-neutral quantal level-1 agent will behave quite aggressively since it believes that the other agent is an uncertain-follower. On the contrary, a risk-neutral quantal level-2 agent will perform more conservatively because it believes that the other agent is aggressively executing a quantal level-1 policy. Fig. 3(b) shows an exemplary trajectory of Case 1. We can see that with two level-1 agents, collision happened due to their aggressiveness, i.e., they both assumed that the other would yield. On the other hand, Fig. 3(d) and Fig. 3(f), respectively, show exemplary trajectories of Case 2 and Case 3 with agents starting from the same locations as in the exemplary trajectory in Fig. 3(b). We can see that in both cases, the two agents managed to avoid collisions. In Case 2, both agents behaved more conservatively, and lead to low efficiency (Fig. 3(d)), while in Case 3, both agents behaved as their opponent expected and generated the most efficient and safe trajectories (Fig. 3(e)). To show the statistical results, we conducted 100 simulations for each case with randomized initial states, and the RS is shown in Fig. 3(a) (green). It is shown that similar to what we have observed in the exemplary trajectories, Case 1 lead to the lowest RS, and Case 3 achieved the highest RS. The RS in Case 2 is in the middle because though both agents behaved conservatively, the wrong beliefs over the other's model may still lead to lower RS compared to Case 3.   Next, we will see how the risk-sensitive CPT model impacts such risk-neutral behaviors. As shown in Fig. 3(a), in Case 1, the risk-sensitive policies help significantly improve the RS of interactions between two quantal level-1 agents, i.e., they performed less aggressively compared to the risk-neutral case. This is because that the CPT model makes the quantal level-1 agents underestimate the possibilities of "yielding" from their opponents, and thus lead to more conservative behaviors with higher RS. Such a conclusion can be verified by comparing the exemplary trajectories shown in Fig. 3(b-c). We can see that compared to the risk-neutral case in Fig. 3(b), under the risksensitive policy, the blue agent decided to yield to the orange one at the fourth step. At the same time, in Case 2 and Case 3, the risk-sensitivity measure makes the quantal level-2 agents overestimate the possibilities of "yielding" from quantal level-1 agents and generate more aggressive behaviors. An exemplary trajectory is shown in Fig. 3(e). We can see that compared to the risk-neutral quantal level-2 agent in Fig. 3(d), the risksensitive quantal level-2 agents waited less steps and lead to collision. Hence, the RS for both Case 2 and Case 3 are reduced compared to the risk-neutral scenarios, as shown in Fig. 3(a).

The inverse problem: reward learning in Bounded Risk-Sensitive Markov Games
In this section, we validate the proposed reward learning algorithm, i.e., Algorithm 3. Namely, we aim to recover the rewards, intelligence levels, and risk parameters of two interactive agents via observed trajectories from them. We also compare the learning performance of the proposed inverse learning algorithm with a baseline inverse reward learning algorithm.

Synthetic expert demonstrations generation
As discussed above, in the inverse problem, we aim to learn the navigation rewards and the CPT parameter γ of both agents, i.e.,ω=(γ, (ω 1 , ω 2 )) with ω 1,2 ∈ R 25 without prior information on their intelligence levels (we need to infer the intelligence levels simultaneously during the learning based on (9)). In order to achieve such a goal, we first need to collect some expert demonstrations. We generate such demonstrations in the navigation environment via the policies derived in the forward problem. Similarly, for generating the demonstrations, we set the parameters in the CPT model as γ 1,2 =0.5 and α 1,2 =0.7, and let agents with mixed levels of intelligence interact with each other using the risk-sensitive quantal level-k policies. We randomized the initial conditions (initial positions and the levels of intelligence) of the agents and collected M =100 such expert demonstrations (i.e., paired navigation trajectories). The approximation parameter κ in Q-value approximation (12) is set to κ = 100 and the learning rate is set to η = 0.0015.

Performance of the proposed reward learning algorithm
Metrics. We evaluate the learning performance via two metrics: 1) the parameter percentage error (PPE), and 2) the policy loss (PL). The PPE of learned parametersω i is defined as |ω i −ω * ,i |/|ω * ,i | withω * ,i being the ground-truth value. The PL denotes the error between the ground truth quantal level-k policies and the policies obtained using the learned reward functions. It is defined as Results. Figure 4(a) and Fig. 4(b) show, respectively, the history of averaged PPE over all parameters and the PL during learning. The solid lines represent the means from 25 trials and the shaded areas are the 95% confidence intervals. The PPEs of all parameters are given in Fig. 4(c). We can see that the proposed inverse learning algorithm can effectively recover both the rewards and the parameter γ in the CPT model for both agents, with all PPEs smaller than 15%. In addition, in Fig. 4(d), we show the identification accuracy of the intelligence levels of experts in the data. More specifically, the identified intelligence level of agent i, i ∈ P, in a demonstration ξ i is given byk i = arg max k∈K P(k|ξ N −1 ). We can see that accuracy ratios of 86% and 92% are achieved for the two agents, respectively. Hence, the results show that the proposed inverse reward learning algorithm can effectively recover rewards, risk-parameters and intelligence levels of interactive agents in BRSMGs.

Performance comparison with a baseline Maximum Entropy IRL algorithm
The baseline IRL algorithm. The baseline inverse learning algorithm is a risk-neutral Maximum Entropy IRL [40] without quantal level-k game settings. Instead, it assumes the agents are following Stackelberg strategies similar to [30]. More specifically, rather than jointly learning rewards for both agents as the proposed algorithm, in the baseline IRL algorithm, we conduct inverse reward learning separately for each agent. In each IRL formulation, the future trajectories of the opponent agent is assumed to be known, i.e., treating the agent as a leader to the opponent agent.
Metrics. In addition to PPE, we also compare the learned rewards with the ground truth rewards using two types of statistical correlations: 1) Pearson's correlation coefficient (PCC) and 2) Spearman's rank correlation coefficient (SCC). PCC characterizes the linear correlation between the ground truth rewards and the recovered rewards (higher PCC represents higher linear correlations), and SCC characterizes the strength and direction of the monotonic relationship between the ground truth rewards and the recovered rewards (higher SCC represents stronger monotonic relationships).
Results. The performance comparison between the proposed approach and the baseline risk-neutral Maximum Entropy IRL is shown in Fig. 5. We can see in Fig. 5(a) that the proposed method can recover more accurate reward values compared to the baseline IRL. This is because the baseline fails to capture the structure biases caused by risk sensitivity and cognitive limitations. Moreover, Fig. 5(b) indicates that the reward values recovered by the proposed method have higher linear correlation and stronger monotonic relationship to the ground-truth rewards. Moreover, with the proposed reward learning algorithm, the risk preference and intelligence levels of both agents can be recovered, which can provide more interpretable human behavior models that can be used to generate more human-like robotic behaviors in human-robot applications.
7 Conclusion. In this paper, we investigated a new type of Markov Game, i.e., the bounded risk-sensitive Markov Game (BRSMG). The proposed game-theoretic framework made the first attempt to capture the bounded intelligence, the cognitive bias, and the irrationality in humans' reasoning processes by integrating the quantal level-k iterative reasoning model and the cumulative prospect theory. Both the forward policy design problem and the inverse reward learning problem have been addressed with not only theoretical analysis but also simulation verification in a navigation scenario.
Simulation results show that the behaviors of agents in BRSMG demonstrate both risk-averse and risk-seeking phenomena, which are consistent with observations from humans. Moreover, in the inverse reward learning task, the proposed bounded risk-sensitive inverse learning algorithm outperforms a baseline risk-neutral inverse learning algorithm by effectively recovering not only the more accurate rewards but also the intelligence levels and the risk-measure parameters of agents given demonstrations of their interactive behaviors. Therefore, the proposed BRSMG framework provides a tool to learn interpretable and heterogeneous human behavior models which are of critical importance in human-robot interactions.

Broader Impact
Humans always interact with each other and make decisions under uncertainties. To better assist humans, a better model to capture their interactive behaviors has attracted a great amount of research efforts in multiple disciplines such as psychology, cognitive science, economics and computer science. The proposed bounded risk-sensitive Markov Game (BRSMG) framework extends classical Markov Game to consider more realistic behaviors of humans such as bounded rationality and risk-sensitive tendency. We investigate how the optimal policies of the BRSMG differ from that of classical Games. Such a framework aims to provide a better modeling of humans' interactive behaviors, so that we can help them make better and more rational investment decisions in markets, we can design more human-like robots to assist humans in a more efficient and friendly manner, and we can help the governments to propose better guidance towards public policies (such as the "shelter-in-place" order and the re-opening guidance for the COVID-19 pandemic) by reasoning about the possible responses from people.
Similar to all other technologies, a better descriptive model for humans' interactive behaviors also face ethical risks. One possible such risk is that the proposed work might make humans more vulnerable to malicious attacks. For example, someone might be able to more accurately infer your personal preferences (rewards and risk models) by observing your behaviors and take advantage of it. Another risk is that though more human-like artificial agents are preferable in terms of efficiency and less tedious work for humans, they might cause more people lose their jobs in more societal sectors.

A Cumulative Prospect Theory
The cumulative prospect theory (CPT) is a non-expected utility theory that describes the risksensitivity of humans' decision-making processes. In this section, we illustrate the probability weighting function and the utility function, specifically when they are using the following functional forms: In Fig. 6 (a), we show an example of the probability weighting function w + , which describes the characteristics of humans to deflate high probabilities and inflate low probabilities. In Fig. 6 (b), we show an example of the utility function u with x 0 = 0 as the reference point.

B The convergence of bounded risk-sensitive policies
In this section, we show the proof of Theorem 1. To begin with, we show two lemmas that facilitate the proof.
Proof. First, it is clear that the above argument holds when a = b. Then, without loss of generality, we assume that a > b. We define a differentiable function f (x) : R → R and f (x) = x α . Then, following the mean value theorem, we can have a). Note that f (c) = αc α−1 ≤ 1 since α ∈ (0, 1] and c > 1. (8) in Section 3.2 of the submitted manuscript is aγ-contraction mapping when R min satisfies R min ≥ 1. That is, for any two value function estimates V i,k 1 and V i,k 2 , we have Proof. Define r i 1,2 (c a −i s,a i ) = R i (s, a i , a −i )+γV i,k 1,2 (s ), then, we can write Note that the inequality (4)(c) holds based on the definition of u i defined in Section 2.3 of the submitted manuscript, namely, Note that the inequality (5a) holds since a −i ∈A −i ρ i (c a −i s,a i ) ≤ 1 as defined in the condition. Proceeding in this way, we conclude that the operator B is aγ-contraction mapping. Now, we restate Theorem 1 in the submitted manuscript and show its proof. Theorem 1. Define the tuple s, a i , a −i :=c a −i s,a i and normalize the transformed probabilities For an arbitrary agent i∈P, if the one-step reward R i is lower-bounded by R min with R min ≥ 1, then ∀s ∈ S and all intelligence levels with k=1, 2, · · · , the dynamic programming problem in (5) of the submitted manuscript can be solved by the following value iteration algorithm: Moreover, as m → ∞, V i,k m+1 converges to the optimal value function V * ,i,k (s). Proof. We prove the theorem by induction.
When k=1, π * ,−i,k−1 =π −i,0 , which is defined in (3) (Definition 2) in the submitted manuscript. Hence, the dynamic programming problem defined in (5) in the submitted manuscript reduces to a single-agent policy optimization problem since the anchoring policy is known and (5) can be expressed as V * ,i,1 (s) = BV * ,i,1 (s). Moreover, (6) guarantees that the assumption a −i ∈A −i ρ i (c a −i s,a i ) ≤ 1 in Lemma 2 holds. Hence, according to Lemma 2, we have and it is clear that V i,1 m → V * ,i,1 as m → ∞. Hence, when k = 1, the algorithm in (7) can solve for the optimal CPT value and the policy π * ,i,1 can be obtained for all i ∈ P. Note that π * ,i,1 depends on i's intelligence level.
Next, we will show that for any k ∈ N + and k > 1, assuming the convergence of V * ,i,k −1 is proved and the policy π * ,i,k −1 is obtained for all i ∈ P, then, similar to (8), we have V i,k m → V * ,i,k as m → ∞.
Again, with the above assumption on V * ,i,k −1 and π * ,i,k −1 , we can see that the dynamic programming problem defined in (5) in the submitted manuscript has been reduced to a single-agent optimal policy optimization problem since the opponent's policy π * ,−i,k −1 is already solved for and thus only depends on agent −i's intelligence level. Moreover, (6) assures that a −i ∈A −i ρ i (c a −i s,a i )=1 for k > 1, satisfying the assumption in Lemma 2. Hence, via the conclusion from Lemma 2 and (8), we can see that V * ,i,k can be solved by the algorithm in (7). The policy π * ,i,k can also be obtained correspondingly for all i ∈ P.
Hence, we have proved that argument in Theorem 1 holds.

C Detailed Derivations in the Inverse Learning Problem
In this section, we show some detailed derivations that needed to compute the gradient of the objective function in (9) in the submitted manuscript.
Note that both w i,+ , p 1 and p 2 depend on the parameter γ i since γ i ∈ω, but only p 1 and p 2 depend on the parameterω −γ i (note thatω −γ i ω \ {γ i }), thus we compute ∂ρ ī ω ∂γ i and ∂ρ i,n ω ∂ω −γ i separately: where ∂p (1,2) ∂ω (c is the partial derivative with respect to the variables p and can be computed straightforwardly based on the functional form w + (p) defined in Section 2.3 of the submitted manuscript (or (1) in this supplementary material). for k > 1 is already known, since we compute ∂π * ,i,k ω ∂ω iteratively and sequentially for k = 1, 2, . . . , k max and for i ∈ P as shown in Algorithm 2 in the submitted manuscript.

D The Convergence of Value Gradient
In this section, we show the proof of Theorem 2. To begin with, we first restate Theorem 2 as follows: Theorem 2. If the one-step reward R i , i ∈ P, is bounded by R i ∈[R min , R max ] satisfying Rmax Moreover, the algorithm converges to ∂V * ,i,k ω /∂ω as m→∞.
Proof. Based on the definition of the CPT measure ((1) in the submitted manuscript), we can write (note that we only consider u + since we consider positive rewards) CPT( x) = ∞ 0 w + P u + ( x) > y dy = ∞ 0 w + P u + (x) > y u + ( ) dy.
Lemma 4. For an arbitrary agent i ∈ P, if i's one-step reward is lower-bounded by R min and upper-bounded by R max , then ∀k ∈ N + , we have V i,k max ≤ Rmax Rmin V i,k min .
Proof. We define θ = Rmax Rmin , then according to (4) in the submitted manuscript, V i,k max can only be achieved if agent i collects the maximum one-step reward at every step. Similarly, V i,k min can only be achieved if agent i collects the minimum one-step reward at every step. Hence, we have V i,k max = CPT π * ,−i,k−1 R max +γCPT π * ,−i,k−1 R max + . . .
Lemma 5. Recall (13)(a) in Theorem 2 in this supporting material, we define an operator ∇BV ,i,k m = V ,i,k m+1 , ∀i, ∈ P, ∀k ∈ N + . Then, the operator ∇B is aγ-contraction mapping if the one-step reward R i is bounded by R i ∈ [R min , R max ] satisfyingγ = Rmax ≤ 0.
Now, we substitute (22) into (21), and then we can write Proceeding in this way, we conclude that the operator ∇B is aγ-contraction mapping, whereγ= Rmax Proof. We first define ∇BV ,i,k m = V ,i,k m+1 , and then Lemma 5 shows that the operator ∇B is a contraction under the given conditions. Then, the statement is proved by induction in a similar way as for Theorem 1, and thus is omitted.