{'fraction_non_alphanumeric': 0.025146400275576988, 'fraction_numerical': 0.0044781260764726145, 'mean_word_length': 4.718064153066967, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Assist Agents by Observing Them

The ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using reinforcement learning usually requires large amounts of online training, which is difficult and costly. On the other hand, offline data about the behavior of the assisted agent might be available, but is non-trivial to take advantage of by methods such as offline reinforcement learning. We introduce methods where the capability to create a representation of the behavior is first pre-trained with offline data, after which only a small amount of interaction data is needed to learn an assisting policy. We test the setting in a gridworld where the helper agent has the capability to manipulate the environment of the assisted artificial agents, and introduce three different scenarios where the assistance considerably improves the performance of the assisted agents.


Introduction
The ability to build an AI system that can assist other AI agents and humans to reach their goals is an obviously important and ambitious goal. There are numerous possible applications that would benefit directly of such a capability. For example, helper agents could be harnessed to guide self-driving cars to allow smoother traffic flow. An example of cooperation with humans could be a helper agent, which is able to infer the goals of a human navigating a web site, and manipulate the website structure dynamically to help the human users reach their goals faster.
In this paper, we consider building such helper agents. We construct settings, where an artificial agent (a helper agent ) observes the behavior of an assisted agent (goal-driven agent ), and learns to perform useful assisting actions by manipulating the environment dynamics.
We formulate the challenge as a meta-reinforcement learning problem. We first train a small population of goal-driven agents to reach their goals as fast as possible. The helper agent observes the goal-driven agents in many instances of the environments, and is trained by meta-reinforcement learning to perform useful assisting actions by receiving a positive reward when the goal-driven agents succeed in their tasks.
As a contribution of our work, we show that similar capabilities to inverse reinforcement learning arise spontaneously as a result of meta-learning. We demonstrate this effect and its usefulness for producing helping actions in three different scenarios: 1. By learning representations of the behavior of the goal-driven agent from the past partial trajectories in an unsupervised way. 2. By learning representations of the behavior of the goal-driven agent from the past partial trajectories in a self-supervised way, by predicting the actions of the goal-driven agent. 3. By learning representations of the behavior of the goal-driven agent from the past partial trajectories in a supervised way, by predicting the goal of the goal-driven agent.

Related work
There is a rich literature on reasoning about the behavior and inferring the goals of other agents [1], which is closely related to our work. Unlike most of the work related to modelling other agents, our specific goal was to build agents which can do assisting actions, based on the inferred representations of the behavior of the other agent. Similar settings have been studied in multi-agent scenarios. As an example, in [5] a two-agent helper-AI scenario is presented, where both agents have the same reward function, but disagree on the transition dynamics of the Markov decision process (MDP). In our setting, in contrast, the action spaces of the agents are different, and the assisted agent is unaware of the assisting agent. The authors in [8] consider learning representations of agent behaviors in multi-agent scenarios. Our approach is similar in that we also learn to represent agents' behaviors, but rather than communicating cooperatively, the helper agent attempts to change the environment in a beneficial manner.
An individual is considered to have a Theory of Mind (ToM), if they impute a mental state to themselves or others [12,19]. An inferred mental state can be used to infer goals and make predictions of behavior of other agents. Our work is inspired by Theory of Mind, as we are training behavioral embeddings of the goal-driven agents by only observing them. Examples of previous work implementing Theory of Mind in a machine learning context include [3], a Bayesian Theory of Mind model, and [20], which is based on deep learning. In our approach, a separate helper agent infers the goal-driven agent behavior and produces behavior embeddings as in [20], which are used for manipulating the environment in a beneficial way.
The problem of learning fast from limited amounts of data can be tackled with meta-learning methods, where the goal is to train a system in such a way that it is capable of performing fast adaptation to new situations [11,22]. We train the helper agent to be able to do few-shot learning of a new goal-driven agent in a new sampled environment by creating a behavior embedding of the agent based on small amounts of observations. Commonly used meta-learning methods include gradient based methods [7] and memory-based methods [6,21,24]. Our meta-learning solution resembles mostly attention-based meta-learning [14], since in our case the behavior embeddings are generated by averaging the state representations over the time steps without recurrent connections.
Our third scenario is partly inspired by the centralized training and decentralized execution (CTDE) concept [10,16], where additional information can be used during the training but not during the execution. Instead of using private information of the goal-driven agent, we use information of the goal-driven agent's goal for pre-training the helper agent policy.
Similar challenges could be solved by transfer learning or by using auxiliary tasks. In transfer learning, the goal is to transfer what has been learned in one task to some other final task, possibly by partially sharing parameters between the models [18,25]. Numerous success cases have demonstrated benefits of transfer learning in the areas of computer vision [17] and natural language processing [2,4]. Our solution is one type of transfer learning, where the goal is specifically to pre-train one specific part of the policy function to allow faster learning for producing assisting actions. When using auxiliary tasks, the common representation among the original task and auxiliary tasks can be improved by adding extra auxiliary heads to the representation [9,13,26]. In our solution, instead of adding an extra auxiliary head to the common representation, the helper agent is pre-trained on the task of making predictions of the goal-driven agents' actions in self-supervised manner or goals in a supervised manner. After completing the pre-training task, the final task training benefits from this ability.
Inverse Reinforcement Learning (IRL) [15] has similar goals, but does so by recovering the unknown reward function of the expert user or any agent it tries to model. The difference in our method is that we do not attempt to recover the reward function but rather model the agent behavior directly by observing the agent actions, and then build the helper policy, which uses the behavior embeddings. This approach is useful when moving to behavior where reward functions are complex or intractable, as often is the case when attempting to model human-like complex behavior.

Setting
We use a meta-reinforcement learning formulation for training both the goaldriven agents and the helper agent. The MDP is defined by a tuple M = S, A, P, R, γ, ρ 0 , H , where S is the state space and A is the action space, γ is the discount factor, ρ 0 : S → [0, 1] is the initial state distribution, P : S × A × S → [0, 1] is the transition function, R : S × A → IR is the reward function and H is the horizon.
The agents maximize the expected sum of discounted rewards over the episodes is the trajectory of states s t and actions a t of the agent at time step t for an episode of length H.
The initial state is sampled from the distribution ρ 0 and the agent j samples the action a t from its policy function π j (a t |s t ). The next state is sampled from the transition dynamics function s t+1 ∼ P(s t+1 |s t , a t ). The agents learn through many episodes. For each episode, a new MDP M i is sampled from a family M of MDPs. The MDPs for the helper agent and the goal-driven agents share the same state space S but have different actions spaces and reward functions denoted as A helper and A goal−driven , and R helper and R goal−driven j . From the helper agent point of view, the behavior of the goal-driven agent is embedded in the transition dynamics of the sampled MDP, and each sampled MDP refers to different task in the meta-reinforcement learning framework. The meta-learning goal is to maximize the expected returns over the task distribution.

The gridworld
We test our architectures in various gridworld scenarios. Every new sampled MDP has a 7 x 7 gridworld environment, which includes four possible goals in random locations, an L-shaped wall in a random location and a goal-driven agent starting in a random position. Each goal-driven agent is trained to navigate to one of the goals with the shortest route. The wall has always the same basic shape, and its location is sampled but restricted so that the long edge of the wall needs to be at least two steps from the border of the gridworld. The left side of Figure 1 illustrates a sampled gridworld environment.
The environment delivers a set of tensors as a state representation, one tensor per one time step. The state representation is a collection of 7x7x6 tensors, where the different channels in one tensor represent goal-driven agent location (1 channel), wall elements (1 channel) and goals (4 channels).
The episodes are divided in two distinct stages. In the first stage, the helper agent observes several partial trajectories of the unassisted goal-driven agent navigating to its goal in different sampled environments. In the second stage, a new environment is sampled and the same goal-driven agent is set in a random location. In the beginning of the second stage, the helper agent observes the state (a single tensor) and will decide on an action based on this observation. The right side of Figure 1 illustrates a situation where the helper agent has acted and opened a shortcut for the goal-driven agent.

The goal-driven agents
The goal-driven agents, which are part of the environment of the helper agent, are trained with the Proximal Policy Optimization (PPO) algorithm [23]. We train four different agent types which have different target goals. The action When the goal-driven agent j follows its policy, it will produce a trajectory consisting of a list of state-action pairs where H is the length of the episode.

The helper agent
Once a population of goal-driven agents exists, the helper agent is trained with the PPO algorithm to infer the agent types by observing them, and to conduct an assisting action by manipulating the transition dynamics (creating a shortcut in a specified location). The action space A helper of the helper agent consists of two actions, either to act or not to act, and the action is performed once at a specified time point depending on the scenario in question, thus the horizon H for the helper agent is a single time step. The reward function R helper , is the same as with the goal-driven agent but augmented with a negative reward for making an action to assist the goal-driven agent. In other words, the helper agent gets a reward when the goal-driven agent succeeds in its task. The helper agent has access only to the partial trajectories of τ j of the goal-driven agents and thus not to their reward function R goal−driven j or policy π goal−driven j .

End-to-end training without agent-specific inference.
We first evaluate the performance of the helper agent in a simplified situation where it does not have access to the past observations of unassisted behavior of the goal-driven agents. The results of this experiment serve as a lower bound for the helper agent performance. The helper agent does not observe the first stage of each episode and needs to decide on its actions only based on the initial state of the second stage. Thus, in this case the helper agent does not have the capability of inferring the behavior of the goal-driven agents.
In this scenario, the helper agent is a neural network with two convolutional layers and one fully-connected layer (see Figure 3, left). More details of the neural network implementation is available in the appendix.
The green line in Figure 2 illustrates the training result in this setting. The helper agent learns to improve its policy slightly despite not being able to infer the agent behavior. In this case, the helper agent tries to find the optimal performance by observing the general behavior of all agent types and by finding a more general policy that does not depend on a single agent's goal. As an example, if all goals are on the other side of the wall, it would be beneficial to open the wall. Likewise, if all goals are on the same side of the wall as the goal-driven agent, it is not useful to open the wall and take the penalty. These two scenarios are illustrated in Figure 3.

Scenario 1: End-to-end training with agent-specific inference.
In the first scenario, the helper agent is exposed to the past observations of unassisted behavior of the goal-driven agents. In the first stage, it now observes four first steps from four different episodes of the same agent but with different sampled layout of the gridworld. By observing these partial trajectories before conducting the assisting action, the helper agent has a chance to learn about the goal-driven agents' goals based on the agents' unassisted behavior. The helper agent architecture is illustrated in Figure 5. During the first stage, the helper agent uses the behavior inference part of the policy. The embeddings of partial episodes are formed by using a convolutional network, followed by mean pooling and a fully connected layer. The behavior embeddings are formed in a similar manner from the episode embeddings but without convolutional layers. During the training, the behavior inference part learns to construct 2-dimensional behavior embeddings by clustering the agents based on their goals, where each cluster contains agents with the same goal ( Figure 4).  The policy head part of the helper agent has the same structure as the simpler case shown in the left side of Figure 3. The task of the policy head is to produce the action distribution but now it additionally receives the behavior embedding from the behavior inference part. The training of this scenario is conducted end-to-end.
The result of this experiment is illustrated by the blue line in Figure 2. The helper agent is capable to use the information produced by the behavior inference part and as a result the performance is improved considerably. Since there is significant variance present in simulations with different seeds, we can assume that the end-to-end training is more sensitive to the initialization than the other scenarios.
In this scenario the helper agent has an initial pre-training phase where it observes multiple partial episodes where the goal-driven agents behave without assistance, simulating a situation where previously collected offline data of the assisted agent behavior is available. The pre-training is conducted in a self-supervised fashion by predicting the action distributions of the goal-driven agents. After the partial rollouts of episodes the distribution over the next action is predicted by using a linear classification layer on top of the 2-dimensional behavior embedding. The cross-entropy loss is calculated between the observed one-hot action and the predicted action distribution. Once the pre-training is complete, the linear classification layer is discarded and the setting is converted to a similar one as in scenario 1.
Pre-training of the behavior inference part with action predictions clearly helps the policy function to learn assisting actions as the learning is faster and there is less variation across different seeds, as is visible by the purple line in Figure 2.

Scenario 3: Supervised pre-training with agent-specific
inference.
In our final scenario the helper agent has again an initial pre-training phase where it observes multiple partial episodes where goal-driven agents behave without assistance. Inspired by the centralized training decentralized execution (CTDE) method [16,10], the helper agent is now provided with the information about the agent's goal in each episode during the pre-training phase, but not during the execution after the pre-training. The pre-training is conducted in a similar manner as in scenario 2 by using the linear classifier on top of the 2-dimensional behavior embedding, and by calculating the cross-entropy loss between the onehot goal label and the predicted goal distribution. In this case, the pre-training has strong supervision, and the helper agent is able to cluster the goal-driven agents based on their goals already during the pre-training phase. This allows faster learning as is visible with the red line in Figure 2.

Summary of results
In order to evaluate the performance of the helper agent in various scenarios, we made additional measurements that are useful for gaining insight of the helper agent performance. We measured the mean return over episodes when the wall was permanently closed, when it was permanently open and the optimal policy by making rollouts from the query state. For each query state, we made a rollout both with wall open and wall closed scenarios. By selecting higher return from these rollouts, we measured the optimal policy. All key results are summarized in Table 1. The results show, that even when the helper agent does not have the possibility to observe the behavior of the goal-driven agent, the performance is improved. The performance is further increased when the helper agent can observe partial episodes of the goal driven agent (scenario 1). The self-supervised pre-training (scenario 2) and supervised pre-training (scenario 3) achieve highest overall performance as they are closest to the optimal performance. Interestingly, the self-supervised scenario reach a similar performance as the supervised pretraining, which suggests that self-supervised method is a potential alternative for assessing behavior when labels are not available for the pre-training.

Discussion
In this paper, we introduced several scenarios, where a helper agent can assist goal-driven agents by manipulating the transition dynamics of the environment. We showed that by using meta-learning, similar capabilities to inverse reinforcement learning arise spontaneously. Our method is also an example how previously collected data can be used to enhance the performance of a helper agent when offline learning methods are not possible. This kind of situation arises if the previously collected data is from the assisted agent behavior, not from a helper agent behavior. The results indicated that by learning to infer the agent's behavior, the helper agent's policy can be significantly improved. Furthermore, the training can be made faster and more robust by a separate pre-training phase. The pre-training can be done either in self-supervised manner by predicting the actions of the assisted agent, or by using additional information about the agents' goals that is available during the training but not during the execution. This kind of scenario is valid when it is of interest to predict a known feature that explains the behavior, such as the goal, skill level, level of stochasticity, level of visibility, or level of memory. An interesting continuation to this work could be extending to modeling more complex human-like behaviors in scenarios that are closer to real-world human-computer interactions (for example, web-site navigation).