End-to-End Egospheric Spatial Memory
  Spatial memory, or the ability to remember and recall specific locations and
objects, is central to autonomous agents' ability to carry out tasks in real
environments. However, most existing artificial memory modules are not very
adept at storing spatial information. We propose a parameter-free module,
Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere
around the agent, enabling expressive 3D representations. ESM can be trained
end-to-end via either imitation or reinforcement learning, and improves both
training efficiency and final performance against other memory baselines on
both drone and manipulator visuomotor control tasks. The explicit egocentric
geometry also enables us to seamlessly combine the learned controller with
other non-learned modalities, such as local obstacle avoidance. We further show
applications to semantic segmentation on the ScanNet dataset, where ESM
naturally combines image-level and map-level inference modalities. Through our
broad set of experiments, we show that ESM provides a general computation graph
for embodied spatial reasoning, and the module forms a bridge between real-time
mapping systems and differentiable memory architectures. Implementation at:
https://github.com/ivy-dl/memory.
