LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of
  Sparse Reward Iterative Tasks
  Reinforcement learning (RL) has shown impressive success in exploring
high-dimensional environments to learn complex tasks, but can often exhibit
unsafe behaviors and require extensive environment interaction when exploration
is unconstrained. A promising strategy for learning in dynamically uncertain
environments is requiring that the agent can robustly return to learned safe
sets, where task success (and therefore safety) can be guaranteed. While this
approach has been successful in low-dimensions, enforcing this constraint in
environments with visual observations is exceedingly challenging. We present a
novel continuous representation for safe sets by framing it as a binary
classification problem in a learned latent space, which flexibly scales to
image observations. We then present a new algorithm, Latent Space Safe Sets
(LS3), which uses this representation for long-horizon tasks with sparse
rewards. We evaluate LS3 on 4 domains, including a challenging sequential
pushing task in simulation and a physical cable routing task. We find that LS3
can use prior task successes to restrict exploration and learn more efficiently
than prior algorithms while satisfying constraints. See
https://tinyurl.com/latent-ss for code and supplementary material.
