Reinforcement Learning from Hierarchical Critics
  In this study, we investigate the use of global information to speed up the
learning process and increase the cumulative rewards of reinforcement learning
(RL) in competition tasks. Within the actor-critic RL, we introduce multiple
cooperative critics from two levels of the hierarchy and propose a
reinforcement learning from hierarchical critics (RLHC) algorithm. In our
approach, each agent receives value information from local and global critics
regarding a competition task and accesses multiple cooperative critics in a
top-down hierarchy. Thus, each agent not only receives low-level details but
also considers coordination from higher levels, thereby obtaining global
information to improve the training performance. Then, we test the proposed
RLHC algorithm against the benchmark algorithm, proximal policy optimisation
(PPO), for two experimental scenarios performed in a Unity environment
consisting of tennis and soccer agents' competitions. The results showed that
RLHC outperforms the benchmark on both competition tasks.
