School of hard knocks: Curriculum analysis for Pommerman with a fixed
  computational budget
  Pommerman is a hybrid cooperative/adversarial multi-agent environment, with
challenging characteristics in terms of partial observability, limited or no
communication, sparse and delayed rewards, and restrictive computational time
limits. This makes it a challenging environment for reinforcement learning (RL)
approaches. In this paper, we focus on developing a curriculum for learning a
robust and promising policy in a constrained computational budget of 100,000
games, starting from a fixed base policy (which is itself trained to imitate a
noisy expert policy). All RL algorithms starting from the base policy use
vanilla proximal-policy optimization (PPO) with the same reward function, and
the only difference between their training is the mix and sequence of opponent
policies. One expects that beginning training with simpler opponents and then
gradually increasing the opponent difficulty will facilitate faster learning,
leading to more robust policies compared against a baseline where all available
opponent policies are introduced from the start. We test this hypothesis and
show that within constrained computational budgets, it is in fact better to
"learn in the school of hard knocks", i.e., against all available opponent
policies nearly from the start. We also include ablation studies where we study
the effect of modifying the base environment properties of ammo and bomb blast
strength on the agent performance.
