Residual Feedback Learning for Contact-Rich Manipulation Tasks with
  Uncertainty
  While classic control theory offers state of the art solutions in many
problem scenarios, it is often desired to improve beyond the structure of such
solutions and surpass their limitations. To this end, residual policy learning
(RPL) offers a formulation to improve existing controllers with reinforcement
learning (RL) by learning an additive "residual" to the output of a given
controller. However, the applicability of such an approach highly depends on
the structure of the controller. Often, internal feedback signals of the
controller limit an RL algorithm to adequately change the policy and, hence,
learn the task. We propose a new formulation that addresses these limitations
by also modifying the feedback signals to the controller with an RL policy and
show superior performance of our approach on a contact-rich peg-insertion task
under position and orientation uncertainty. In addition, we use a recent
Cartesian impedance control architecture as the control framework which can be
available to us as a black-box while assuming no knowledge about its
input/output structure, and show the difficulties of standard RPL. Furthermore,
we introduce an adaptive curriculum for the given task to gradually increase
the task difficulty in terms of position and orientation uncertainty. A video
showing the results can be found at https://youtu.be/SAZm_Krze7U .
