Learning Altruistic Behaviours in Reinforcement Learning without
  External Rewards
  Can artificial agents learn to assist others in achieving their goals without
knowing what those goals are? Generic reinforcement learning agents could be
trained to behave altruistically towards others by rewarding them for
altruistic behaviour, i.e., rewarding them for benefiting other agents in a
given situation. Such an approach assumes that other agents' goals are known so
that the altruistic agent can cooperate in achieving those goals. However,
explicit knowledge of other agents' goals is often difficult to acquire. In the
case of human agents, their goals and preferences may be difficult to express
fully; they might be ambiguous or even contradictory. Thus, it is beneficial to
develop agents that do not depend on external supervision and learn altruistic
behaviour in a task-agnostic manner. We propose to act altruistically towards
other agents by giving them more choice and allowing them to achieve their
goals better. Some concrete examples include opening a door for others or
safeguarding them to pursue their objectives without interference. We formalize
this concept and propose an altruistic agent that learns to increase the
choices another agent has by preferring to maximize the number of states that
the other agent can reach in its future. We evaluate our approach in three
different multi-agent environments where another agent's success depends on
altruistic behaviour. Finally, we show that our unsupervised agents can perform
comparably to agents explicitly trained to work cooperatively, in some cases
even outperforming them.
