Learning Reciprocity in Complex Sequential Social Dilemmas
  Reciprocity is an important feature of human social interaction and underpins
our cooperative nature. What is more, simple forms of reciprocity have proved
remarkably resilient in matrix game social dilemmas. Most famously, the
tit-for-tat strategy performs very well in tournaments of Prisoner's Dilemma.
Unfortunately this strategy is not readily applicable to the real world, in
which options to cooperate or defect are temporally and spatially extended.
Here, we present a general online reinforcement learning algorithm that
displays reciprocal behavior towards its co-players. We show that it can induce
pro-social outcomes for the wider group when learning alongside selfish agents,
both in a $2$-player Markov game, and in $5$-player intertemporal social
dilemmas. We analyse the resulting policies to show that the reciprocating
agents are strongly influenced by their co-players' behavior.
