Correcting Robot Plans with Natural Language Feedback
  When humans design cost or goal specifications for robots, they often produce
specifications that are ambiguous, underspecified, or beyond planners' ability
to solve. In these cases, corrections provide a valuable tool for
human-in-the-loop robot control. Corrections might take the form of new goal
specifications, new constraints (e.g. to avoid specific objects), or hints for
planning algorithms (e.g. to visit specific waypoints). Existing correction
methods (e.g. using a joystick or direct manipulation of an end effector)
require full teleoperation or real-time interaction. In this paper, we explore
natural language as an expressive and flexible tool for robot correction. We
describe how to map from natural language sentences to transformations of cost
functions. We show that these transformations enable users to correct goals,
update robot motions to accommodate additional user preferences, and recover
from planning errors. These corrections can be leveraged to get 81% and 93%
success rates on tasks where the original planner failed, with either one or
two language corrections. Our method makes it possible to compose multiple
constraints and generalizes to unseen scenes, objects, and sentences in
simulated environments and real-world environments.
