Learning Synthetic Environments and Reward Networks for Reinforcement
  Learning
  We introduce Synthetic Environments (SEs) and Reward Networks (RNs),
represented by neural networks, as proxy environment models for training
Reinforcement Learning (RL) agents. We show that an agent, after being trained
exclusively on the SE, is able to solve the corresponding real environment.
While an SE acts as a full proxy to a real environment by learning about its
state dynamics and rewards, an RN is a partial proxy that learns to augment or
replace rewards. We use bi-level optimization to evolve SEs and RNs: the inner
loop trains the RL agent, and the outer loop trains the parameters of the SE /
RN via an evolution strategy. We evaluate our proposed new concept on a broad
range of RL algorithms and classic control environments. In a one-to-one
comparison, learning an SE proxy requires more interactions with the real
environment than training agents only on the real environment. However, once
such an SE has been learned, we do not need any interactions with the real
environment to train new agents. Moreover, the learned SE proxies allow us to
train agents with fewer interactions while maintaining the original task
performance. Our empirical results suggest that SEs achieve this result by
learning informed representations that bias the agents towards relevant states.
Moreover, we find that these proxies are robust against hyperparameter
variation and can also transfer to unseen agents.
