Variable Impedance Control in End-Effector Space: An Action Space for
  Reinforcement Learning in Contact-Rich Tasks
  Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded
impressive results in recent years. While many studies in RL focus on varying
the observation space or reward model, few efforts focused on the choice of
action space (e.g. joint or end-effector space, position, velocity, etc.).
However, studies in robot motion control indicate that choosing an action space
that conforms to the characteristics of the task can simplify exploration and
improve robustness to disturbances. This paper studies the effect of different
action spaces in deep RL and advocates for Variable Impedance Control in
End-effector Space (VICES) as an advantageous action space for constrained and
contact-rich tasks. We evaluate multiple action spaces on three prototypical
manipulation tasks: Path Following (task with no contact), Door Opening (task
with kinematic constraints), and Surface Wiping (task with continuous contact).
We show that VICES improves sample efficiency, maintains low energy
consumption, and ensures safety across all three experimental setups. Further,
RL policies learned with VICES can transfer across different robot models in
simulation, and from simulation to real for the same robot. Further information
is available at https://stanfordvl.github.io/vices.
