Common Information based Approximate State Representations in
  Multi-Agent Reinforcement Learning
  Due to information asymmetry, finding optimal policies for Decentralized
Partially Observable Markov Decision Processes (Dec-POMDPs) is hard with the
complexity growing doubly exponentially in the horizon length. The challenge
increases greatly in the multi-agent reinforcement learning (MARL) setting
where the transition probabilities, observation kernel, and reward function are
unknown. Here, we develop a general compression framework with approximate
common and private state representations, based on which decentralized policies
can be constructed. We derive the optimality gap of executing dynamic
programming (DP) with the approximate states in terms of the approximation
error parameters and the remaining time steps. When the compression is exact
(no error), the resulting DP is equivalent to the one in existing work. Our
general framework generalizes a number of methods proposed in the literature.
The results shed light on designing practically useful deep-MARL network
structures under the "centralized learning distributed execution" scheme.
