Visually Grounding Language Instruction for History-Dependent
  Manipulation
  This paper emphasizes the importance of a robot's ability to refer to its
task history, especially when it executes a series of pick-and-place
manipulations by following language instructions given one by one. The
advantage of referring to the manipulation history can be categorized into two
folds: (1) the language instructions omitting details but using expressions
referring to the past can be interpreted, and (2) the visual information of
objects occluded by previous manipulations can be inferred. For this, we
introduce a history-dependent manipulation task which objective is to visually
ground a series of language instructions for proper pick-and-place
manipulations by referring to the past. We also suggest a relevant dataset and
model which can be a baseline, and show that our model trained with the
proposed dataset can also be applied to the real world based on the CycleGAN.
Our dataset and code are publicly available on the project website:
https://sites.google.com/view/history-dependent-manipulation.
