StructFormer: Learning Spatial Structure for Language-Guided Semantic
  Rearrangement of Novel Objects
  Geometric organization of objects into semantically meaningful arrangements
pervades the built world. As such, assistive robots operating in warehouses,
offices, and homes would greatly benefit from the ability to recognize and
rearrange objects into these semantically meaningful structures. To be useful,
these robots must contend with previously unseen objects and receive
instructions without significant programming. While previous works have
examined recognizing pairwise semantic relations and sequential manipulation to
change these simple relations none have shown the ability to arrange objects
into complex structures such as circles or table settings. To address this
problem we propose a novel transformer-based neural network, StructFormer,
which takes as input a partial-view point cloud of the current object
arrangement and a structured language command encoding the desired object
configuration. We show through rigorous experiments that StructFormer enables a
physical robot to rearrange novel objects into semantically meaningful
structures with multi-object relational constraints inferred from the language
command.
