Blocks Assemble! Learning to Assemble with Large-Scale Structured
  Reinforcement Learning
  Assembly of multi-part physical structures is both a valuable end product for
autonomous robotics, as well as a valuable diagnostic task for open-ended
training of embodied intelligent agents. We introduce a naturalistic
physics-based environment with a set of connectable magnet blocks inspired by
children's toy kits. The objective is to assemble blocks into a succession of
target blueprints. Despite the simplicity of this objective, the compositional
nature of building diverse blueprints from a set of blocks leads to an
explosion of complexity in structures that agents encounter. Furthermore,
assembly stresses agents' multi-step planning, physical reasoning, and bimanual
coordination. We find that the combination of large-scale reinforcement
learning and graph-based policies -- surprisingly without any additional
complexity -- is an effective recipe for training agents that not only
generalize to complex unseen blueprints in a zero-shot manner, but even operate
in a reset-free setting without being trained to do so. Through extensive
experiments, we highlight the importance of large-scale training, structured
representations, contributions of multi-task vs. single-task learning, as well
as the effects of curriculums, and discuss qualitative behaviors of trained
agents.
