Hierarchical RNNs-Based Transformers MADDPG for Mixed
  Cooperative-Competitive Environments
  At present, attention mechanism has been widely applied to the fields of deep
learning models. Structural models that based on attention mechanism can not
only record the relationships between features position, but also can measure
the importance of different features based on their weights. By establishing
dynamically weighted parameters for choosing relevant and irrelevant features,
the key information can be strengthened, and the irrelevant information can be
weakened. Therefore, the efficiency of deep learning algorithms can be
significantly elevated and improved. Although transformers have been performed
very well in many fields including reinforcement learning, there are still many
problems and applications can be solved and made with transformers within this
area. MARL (known as Multi-Agent Reinforcement Learning) can be recognized as a
set of independent agents trying to adapt and learn through their way to reach
the goal. In order to emphasize the relationship between each MDP decision in a
certain time period, we applied the hierarchical coding method and validated
the effectiveness of this method. This paper proposed a hierarchical
transformers MADDPG based on RNN which we call it Hierarchical RNNs-Based
Transformers MADDPG(HRTMADDPG). It consists of a lower level encoder based on
RNNs that encodes multiple step sizes in each time sequence, and it also
consists of an upper sequence level encoder based on transformer for learning
the correlations between multiple sequences so that we can capture the causal
relationship between sub-time sequences and make HRTMADDPG more efficient.
