Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D
  Environments
  Curiosity is a general method for augmenting an environment reward with an
intrinsic reward, which encourages exploration and is especially useful in
sparse reward settings. As curiosity is calculated using next state prediction
error, the type of state encoding used has a large impact on performance.
Random features and inverse-dynamics features are generally preferred over VAEs
based on previous results from Atari and other mostly 2D environments. However,
unlike VAEs, they may not encode sufficient information for optimal behaviour,
which becomes increasingly important as environments become more complex. In
this paper, we use the sparse reward 3D physics environment Animal-AI, to
demonstrate how a fixed $\beta$-VAE encoding can be used effectively with
curiosity. We combine this with curriculum learning to solve the previously
unsolved exploration intensive detour tasks while achieving 22\% gain in sample
efficiency on the training curriculum against the next best encoding. We also
corroborate the results on Atari Breakout, with our custom encoding
outperforming random features and inverse-dynamics features.
