Developing cooperative policies for multi-stage reinforcement learning tasks
Many hierarchical reinforcement learning algorithms utilise a series of
independent skills as a basis to solve tasks at a higher level of reasoning.
These algorithms don't consider the value of using skills that are cooperative
instead of independent. This paper proposes the Cooperative Consecutive
Policies (CCP) method of enabling consecutive agents to cooperatively solve
long time horizon multi-stage tasks. This method is achieved by modifying the
policy of each agent to maximise both the current and next agent's critic.
Cooperatively maximising critics allows each agent to take actions that are
beneficial for its task as well as subsequent tasks. Using this method in a
multi-room maze domain and a peg in hole manipulation domain, the cooperative
policies were able to outperform a set of naive policies, a single agent
trained across the entire domain, as well as another sequential HRL algorithm.